Re: [Numpy-discussion] Release tags

[Julian Taylor]

probably the git quirk of not pulling tags by default, you need to run:
git pull --tags

On 23.03.2017 19:10, Matti Picus wrote:
> I am searching for the exact git hash corresponding to numpy 1.12.1
> 
> Looking at the result of `git tag -l` yielded tags for v1.11.0, but
> nothing higher.
> 
> Also the documentation site
> https://docs.scipy.org/doc/numpy/release.html does not seem to list
> 1.12.1, but I guess that is a separate issue?
> 
> Matti
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] caching large allocations on gnu/linux

[Julian Taylor]

On 13.03.2017 19:54, Francesc Alted wrote:
> 2017-03-13 18:11 GMT+01:00 Julian Taylor <jtaylor.deb...@googlemail.com
> <mailto:jtaylor.deb...@googlemail.com>>:
> 
> On 13.03.2017 16:21, Anne Archibald wrote:
> >
> >
> > On Mon, Mar 13, 2017 at 12:21 PM Julian Taylor
> > <jtaylor.deb...@googlemail.com
> <mailto:jtaylor.deb...@googlemail.com>
> <mailto:jtaylor.deb...@googlemail.com
> <mailto:jtaylor.deb...@googlemail.com>>>
> > wrote:
> >
> > Should it be agreed that caching is worthwhile I would propose a 
> very
> > simple implementation. We only really need to cache a small handful 
> of
> > array data pointers for the fast allocate deallocate cycle that 
> appear
> > in common numpy usage.
> > For example a small list of maybe 4 pointers storing the 4 largest
> > recent deallocations. New allocations just pick the first memory 
> block
> > of sufficient size.
> > The cache would only be active on systems that support MADV_FREE 
> (which
> > is linux 4.5 and probably BSD too).
> >
> > So what do you think of this idea?
> >
> >
> > This is an interesting thought, and potentially a nontrivial speedup
> > with zero user effort. But coming up with an appropriate caching policy
> > is going to be tricky. The thing is, for each array, numpy grabs a block
> > "the right size", and that size can easily vary by orders of magnitude,
> > even within the temporaries of a single expression as a result of
> > broadcasting. So simply giving each new array the smallest cached block
> > that will fit could easily result in small arrays in giant allocated
> > blocks, wasting non-reclaimable memory.  So really you want to recycle
> > blocks of the same size, or nearly, which argues for a fairly large
> > cache, with smart indexing of some kind.
> >
> 
> The nice thing about MADV_FREE is that we don't need any clever cache.
> The same process that marked the pages free can reclaim them in another
> allocation, at least that is what my testing indicates it allows.
> So a small allocation getting a huge memory block does not waste memory
> as the top unused part will get reclaimed when needed, either by numpy
> itself doing another allocation or a different program on the system.
> 
> 
> ​Well, what you say makes a lot of sense to me, so if you have tested
> that then I'd say that this is worth a PR and see how it works on
> different workloads​.
>  
> 
> 
> An issue that does arise though is that this memory is not available for
> the page cache used for caching on disk data. A too large cache might
> then be detrimental for IO heavy workloads that rely on the page cache.
> 
> 
> ​Yeah.  Also, memory mapped arrays use the page cache intensively, so we
> should test this use case​ and see how the caching affects memory map
> performance.
>  
> 
> So we might want to cap it to some max size, provide an explicit on/off
> switch and/or have numpy IO functions clear the cache.
> 
> 
> ​Definitely​ dynamically
>  allowing the disabling
> ​this feature would be desirable.  That would provide an easy path for
> testing how it affects performance.  Would that be feasible?
> 
> 


I have created a PR with such a simple cache implemented:
https://github.com/numpy/numpy/pull/8783

This sets the max amount of memory pointers to save and returns the old
value:
np.core.multiarray.set_memory_cache_size(4)
On system where it works it return a value greater 0 (4 currently).
The size of the cache in bytes is currently unbounded.
Setting the value to 0 clears and disables the cache.

You should probably not expect too large performance improvements. It
will only have an effect in applications that have measurable page
faulting overhead which only happens if you have lots of relatively
short operations that create copies of large arrays. So mostly
operations with temporaries and maybe some indexing operations.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] caching large allocations on gnu/linux

[Julian Taylor]

On 13.03.2017 16:21, Anne Archibald wrote:
> 
> 
> On Mon, Mar 13, 2017 at 12:21 PM Julian Taylor
> <jtaylor.deb...@googlemail.com <mailto:jtaylor.deb...@googlemail.com>>
> wrote:
> 
> Should it be agreed that caching is worthwhile I would propose a very
> simple implementation. We only really need to cache a small handful of
> array data pointers for the fast allocate deallocate cycle that appear
> in common numpy usage.
> For example a small list of maybe 4 pointers storing the 4 largest
> recent deallocations. New allocations just pick the first memory block
> of sufficient size.
> The cache would only be active on systems that support MADV_FREE (which
> is linux 4.5 and probably BSD too).
> 
> So what do you think of this idea?
> 
> 
> This is an interesting thought, and potentially a nontrivial speedup
> with zero user effort. But coming up with an appropriate caching policy
> is going to be tricky. The thing is, for each array, numpy grabs a block
> "the right size", and that size can easily vary by orders of magnitude,
> even within the temporaries of a single expression as a result of
> broadcasting. So simply giving each new array the smallest cached block
> that will fit could easily result in small arrays in giant allocated
> blocks, wasting non-reclaimable memory.  So really you want to recycle
> blocks of the same size, or nearly, which argues for a fairly large
> cache, with smart indexing of some kind.
> 

The nice thing about MADV_FREE is that we don't need any clever cache.
The same process that marked the pages free can reclaim them in another
allocation, at least that is what my testing indicates it allows.
So a small allocation getting a huge memory block does not waste memory
as the top unused part will get reclaimed when needed, either by numpy
itself doing another allocation or a different program on the system.

An issue that does arise though is that this memory is not available for
the page cache used for caching on disk data. A too large cache might
then be detrimental for IO heavy workloads that rely on the page cache.
So we might want to cap it to some max size, provide an explicit on/off
switch and/or have numpy IO functions clear the cache.



signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] caching large allocations on gnu/linux

[Julian Taylor]

Hi,
As numpy often allocates large arrays and one factor in its performance
is faulting memory from the kernel to the process. This has some cost
that is relatively significant. For example in this operation on large
arrays it accounts for 10-15% of the runtime:

import numpy as np
a = np.ones(1000)
b = np.ones(1000)

%timeit (a * b)**2 + 3

  54.45%  ipython  umath.so [.] sse2_binary_multiply_DOUBLE
  20.43%  ipython  umath.so [.] DOUBLE_add
  16.66%  ipython  [kernel.kallsyms][k] clear_page

The reason for this is that the glibc memory allocator uses memory
mapping for large allocations instead of reusing already faulted memory.
The reason for this is to return memory back to the system immediately
when it is free to keep the whole system more robust.
This makes a lot of sense in general but not so much for many numerical
applications that often are the only thing running.
But despite if have been shown in an old paper that caching memory in
numpy speeds up many applications, numpys usage is diverse so we
couldn't really diverge from the glibc behaviour.

Until Linux 4.5 added support for madvise(MADV_FREE). This flag of the
madvise syscall tells the kernel that a piece of memory can be reused by
other processes if there is memory pressure. Should another process
claim the memory and the original process want to use it again the
kernel will fault new memory into its place so it behaves exactly as if
it was just freed regularly.
But when no other process claims the memory and the original process
wants to reuse it, the memory do not need to be faulted again.

So effectively this flag allows us to cache memory inside numpy that can
be reused by the rest of the system if required.
Doing gives the expected speedup in the above example.

An issue is that the memory usage of numpy applications will seem to
increase. The memory that is actually free will still show up in the
usual places you look at memory usage. Namely the resident memory usage
of the process in top, /proc etc. The usage will only go down when the
memory is actually needed by other processes.
This probably would break some of the memory profiling tools so probably
we need a switch to disable the caching for the profiling tools to use.
Another concern is that using this functionality is actually the job of
the system memory allocator but I had a look at glibcs allocator and it
does not look like an easy job to make good use of MADV_FREE
retroactively, so I don't expect this to happen anytime soon.


Should it be agreed that caching is worthwhile I would propose a very
simple implementation. We only really need to cache a small handful of
array data pointers for the fast allocate deallocate cycle that appear
in common numpy usage.
For example a small list of maybe 4 pointers storing the 4 largest
recent deallocations. New allocations just pick the first memory block
of sufficient size.
The cache would only be active on systems that support MADV_FREE (which
is linux 4.5 and probably BSD too).

So what do you think of this idea?

cheers,
Julian



signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Could we simplify backporting?

[Julian Taylor]

On 24.02.2017 16:00, Evgeni Burovski wrote:
>> I really don't like the double work and the large amount of noise coming
>> from backporting every other PR to NumPy very quickly. For SciPy the policy
>> is:
>>   - anyone can set the "backport-candidate" label
>>   - the release manager backports, usually a bunch in one go
>>   - only important fixes get backported (involves some judging, but things
>> like silencing warnings, doc fixes, etc. are not important enough)
>>
>> This works well, and I'd hope that we can make the NumPy approach similar.
> 
> 
> Just to add to what Ralf is saying:
> 
> * people sometimes send PRs against maintenance branches instead of
> master. In scipy we just label these as backport-candidate, and then
> the RM sorts them out: which ones to forward port and which ones to
> backport. This works OK on scipy scale (I had just trawled though a
> half dozen or so). If numpy needs more backport activity, it might
> make sense to have separate labels for backport-candidate and
> needs-forward-port.
> 
> * A while ago Julian was advocating for some git magic of basing PRs
> on the common merge base for master and maintenance branches, so that
> a commit can be merged directly without a cherry-pick (I think). This
> seems to be beyond a common git-fu (beyond mine for sure!). What I did
> in scipy, I just edited the commit messages after cherry-picking to
> add a reference of the original PR a commit was cherry-picked from.
> 

from the bugfix branch:

git rebase --onto $(git merge-base master maintenance) HEAD^
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] offset in fill diagonal

[Julian Taylor]

On 21.01.2017 16:10, josef.p...@gmail.com wrote:
> Is there a simple way to fill in diagonal elements in an array for other
> than main diagonal?
> 
> As far as I can see, the diagxxx functions that have offset can only
> read and not inplace modify, and the functions for modifying don't have
> offset and only allow changing the main diagonal.
> 
> Usecase: creating banded matrices (2-D arrays) similar to toeplitz.
> 

you can construct index arrays or boolean masks to index using the
np.tri* functions.
e.g.

a = np.arange(5*5).reshape(5,5)
band = np.tri(5, 5, 1, dtype=np.bool) & ~np.tri(5, 5, -2, dtype=np.bool)
a[band] = -1
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy 1.12.0 release

[Julian Taylor]

The version of gcc used will make a large difference in some places.
E.g. the AVX2 integer ufuncs require something around 4.5 to work and in
general the optimization level of gcc has improved greatly since the
clang competition showed up around that time. centos 5 has 4.1 which is
really ancient.
I though the wheels used newer gccs also on centos 5?

On 18.01.2017 08:27, Nathan Goldbaum wrote:
> I've seen reports on the anaconda mailing list of people seeing similar
> speed ups when they compile e.g. Numpy with a recent gcc. Anaconda has
> the same issue as manylinux in that they need to use versions of GCC
> available on CentOS 5.
> 
> Given the upcoming official EOL for CentOS5, it might make sense to
> think about making a pep for a CentOS 6-based manylinux2 docker image,
> which will allow compiling with a newer GCC.
> 
> On Tue, Jan 17, 2017 at 9:15 PM Jerome Kieffer  > wrote:
> 
> On Tue, 17 Jan 2017 08:56:42 -0500
> 
> Neal Becker > wrote:
> 
> 
> 
> > I've installed via pip3 on linux x86_64, which gives me a wheel.  My
> 
> > question is, am I loosing significant performance choosing this
> pre-built
> 
> > binary vs. compiling myself?  For example, my processor might have
> some more
> 
> > features than the base version used to build wheels.
> 
> 
> 
> Hi,
> 
> 
> 
> I have done some benchmarking (%timeit) for my code running in a
> 
> jupyter-notebook within a venv installed with pip+manylinux wheels
> 
> versus ipython and debian packages (on the same computer).
> 
> I noticed the debian installation was ~20% faster.
> 
> 
> 
> I did not investigate further if those 20% came from the manylinux (I
> 
> suspect) or from the notebook infrastructure.
> 
> 
> 
> HTH,
> 
> --
> 
> Jérôme Kieffer
> 
> 
> 
> ___
> 
> NumPy-Discussion mailing list
> 
> NumPy-Discussion@scipy.org 
> 
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Does numpy.bincount support numpy.float128 type weights?

[Julian Taylor]

Also note that float128 is rarely what you want.
It is not a quad precision value, it maps to C long double which is 80
bit on x86 and less on stuff like arm.

On 30.11.2016 22:59, Nathan Goldbaum wrote:
> I think this is a deficiency in the current implementation of bincount,
> which always casts the weights to float64. This WIP pull request should
> probably fix it:
> 
> https://github.com/numpy/numpy/pull/7464
> 
> On Wed, Nov 30, 2016 at 3:54 PM, Wei, Huayi  > wrote:
> 
> Hi, There,
> 
> Here is a sample code using `numpy.bincount`
> 
> import numpy as np
> a = np.array([1.0, 2.0, 3.0], dtype=np.float128)
> b = np.array([1, 2, 0], dtype=np.int )
> c = np.bincount(b, weights=a)
> 
> If run it, I get the following error report:
> 
> > 1 c = np.bincount(b, weights=a)
> TypeError: Cannot cast array data from dtype('float128') to
> dtype('float64') according to the rule 'safe'
> 
> Is it a bug of `np.bincount`? Does there exist any similar function
> which I can use to do the similar thing with numpy.float128 type
> weights?
> 
> Best
> 
> Huayi
> 
> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org 
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ufunc for sum of squared difference

[Julian Taylor]

here is an old unfinished PR adding max_abs which has a similar purpose
as yours:
https://github.com/numpy/numpy/pull/5509

So far I know data is set during runtime construction and is part of the
ufunc object, so it is not really very suitable to pass in constants on
call.
Its intended purpose is pass in functions to be called in generic loops
like PyUFunc_d_d.

Having an option to mark arguments of a ufunc as special in reductions
could be useful, e.g. it would allow a potential
(fused-)multiply-and-add ufunc to be used to implement a weighted sum.

On 11.11.2016 17:25, Matthew Harrigan wrote:
> I started a ufunc to compute the sum of square differences here
> .  It
> is about 4x faster and uses half the memory compared to
> np.sum(np.square(x-c)).  This could significantly speed up common
> operations like std and var (where c=np.mean(x).  It faster because its
> a single pass instead of 3, and also because the inner loop is
> specialized for the common reduce case, which might be an optimization
> considering more generally.
> 
> I think I have answered my question #2&3 above.
> 
> Can someone please point me to an example where "data" was used in a
> ufunc inner loop?  How can that value be set at runtime?  Thanks
> 
> On Fri, Nov 4, 2016 at 5:33 PM, Sebastian Berg
> > wrote:
> 
> On Fr, 2016-11-04 at 15:42 -0400, Matthew Harrigan wrote:
> > I didn't notice identity before.  Seems like frompyfunc always sets
> > it to None.  If it were zero maybe it would work as desired here.
> >
> > In the writing your own ufunc doc, I was wondering if the pointer to
> > data could be used to get a constant at runtime.  If not, what could
> > that be used for?
> > static void double_logit(char **args, npy_intp *dimensions,
> > npy_intp* steps, void* data)
> > Why would the numerical accuracy be any different?  The subtraction
> > and square operations look identical and I thought np.sum just calls
> > np.add.reduce, so the reduction step uses the same code and would
> > therefore have the same accuracy.
> >
> 
> Sorry, did not read it carefully, I guess `c` is the mean, so you are
> doing the two pass method.
> 
> - Sebastian
> 
> 
> > Thanks
> >
> > On Fri, Nov 4, 2016 at 1:56 PM, Sebastian Berg  > s.net > wrote:
> > > On Fr, 2016-11-04 at 13:11 -0400, Matthew Harrigan wrote:
> > > > I was reading this and got thinking about if a ufunc could
> > > compute
> > > > the sum of squared differences in a single pass without a
> > > temporary
> > > > array.  The python code below demonstrates a possible approach.
> > > >
> > > > import numpy as np
> > > > x = np.arange(10)
> > > > c = 1.0
> > > > def add_square_diff(x1, x2):
> > > > return x1 + (x2-c)**2
> > > > ufunc = np.frompyfunc(add_square_diff, 2, 1)
> > > > print(ufunc.reduce(x) - x[0] + (x[0]-c)**2)
> > > > print(np.sum(np.square(x-c)))
> > > >
> > > > I have (at least) 4 questions:
> > > > 1. Is it possible to pass run time constants to a ufunc written
> > > in C
> > > > for use in its inner loop, and if so how?
> > >
> > > I don't think its anticipated, since a ufunc could in most cases
> > > use a
> > > third argument, but a 3 arg ufunc can't be reduced. Not sure if
> > > there
> > > might be some trickery possible.
> > >
> > > > 2. Is it possible to pass an initial value to reduce to avoid the
> > > > clean up required for the first element?
> > >
> > > This is the identity normally. But the identity can only be 0, 1 or
> > > -1
> > > right now I think. The identity is what the output array gets
> > > initialized with (which effectively makes it the first value passed
> > > into the inner loop).
> > >
> > > > 3. Does that ufunc work, or are there special cases which cause
> > > it to
> > > > fall apart?
> > > > 4. Would a very specialized ufunc such as this be considered for
> > > > incorporating in numpy since it would help reduce time and memory
> > > of
> > > > functions already in numpy?
> > > >
> > >
> > > Might be mixing up things, however, IIRC the single pass approach
> > > has a
> > > bad numerical accuracy, so that I doubt that it is a good default
> > > algorithm.
> > >
> > > - Sebastian
> > >
> > >
> > > > Thank you,
> > > > Matt
> > > > ___
> > > > NumPy-Discussion mailing list
> > > > NumPy-Discussion@scipy.org 
> > > > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> > > 

Re: [Numpy-discussion] missing from contributor list?

[Julian Taylor]

you can add multiple email addresses to your github settings so commits 
are properly attributed to your account.



On 11/03/2016 11:10 AM, m...@telenczuk.pl wrote:

Hi,

I had the same problem when I changed my email address. It seems that github 
matches user profiles to commits using the email address, so you have to make 
sure that the email you sign your commits with is registered in your github 
account.

Cheers,

Bartosz

Charles R Harris wrote:

On Wed, Nov 2, 2016 at 4:38 PM, Sturla Molden 
wrote:


Why am I missing from the contributor hist here?

https://github.com/numpy/numpy/blob/master/numpy/_
build_utils/src/apple_sgemv_fix.c




You still show up in the commit log of if you follow the file
```
git log --follow numpy/_build_utils/apple_accelerate.py
```

So I have to agree with others that the problem is on the github end.

Chuck



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] missing from contributor list?

[Julian Taylor]

On 02.11.2016 23:38, Sturla Molden wrote:
> Why am I missing from the contributor hist here?
> 
> https://github.com/numpy/numpy/blob/master/numpy/_build_utils/src/apple_sgemv_fix.c
> 
> 

Probably because the file was moved and github does not track this. This
is something you should ask gihub about and not this list.

fwiw. git blame -C still shows it

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Intel random number package

[Julian Taylor]

As I understand it the wiki is talking about including code in 
numpy/scipy itself, all code in numpy and scipy must be permissively 
licensed so it is easy to reason about when building your binaries.


The license of the binaries produced from the code is a different 
matter, which at that time didn't really exist as we didn't distribute 
binaries at all (except for windows).


A GPL licensed binary containing numpy is perfectly compatible with 
SciPy. It may not be compatible with some other component which has an 
actually incompatible license (e.g. anything you cannot distribute the 
source of as required by the GPL).
I it is not numpy that is GPL licensed it is the restriction of another 
component in the binary distribution that makes the full product adhere 
to the most restrictive license
But numpy itself is always permissive, the distributor can always build 
a permissive numpy binary without the viral component in it.



On 10/27/2016 05:42 PM, Robert McLeod wrote:

Releasing NumPy under GPL would make it incompatible with SciPy, which
may be _slightly_ inconvenient to the scientific Python community:

https://scipy.github.io/old-wiki/pages/License_Compatibility.html

https://mail.scipy.org/pipermail/scipy-dev/2013-August/019149.html

Robert

On Thu, Oct 27, 2016 at 5:14 PM, Julian Taylor
<jtaylor.deb...@googlemail.com <mailto:jtaylor.deb...@googlemail.com>>
wrote:

On 10/27/2016 04:52 PM, Todd wrote:

On Thu, Oct 27, 2016 at 10:43 AM, Julian Taylor
<jtaylor.deb...@googlemail.com
<mailto:jtaylor.deb...@googlemail.com>
<mailto:jtaylor.deb...@googlemail.com
<mailto:jtaylor.deb...@googlemail.com>>>
wrote:

On 10/27/2016 04:30 PM, Todd wrote:

On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers
<ralf.gomm...@gmail.com <mailto:ralf.gomm...@gmail.com>
<mailto:ralf.gomm...@gmail.com <mailto:ralf.gomm...@gmail.com>>
<mailto:ralf.gomm...@gmail.com
<mailto:ralf.gomm...@gmail.com> <mailto:ralf.gomm...@gmail.com
<mailto:ralf.gomm...@gmail.com>>>>
wrote:


On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr
<oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>
<mailto:oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>>
<mailto:oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>
<mailto:oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>>>> wrote:

Please see responses inline.



*From:*NumPy-Discussion
[mailto:numpy-discussion-boun...@scipy.org
<mailto:numpy-discussion-boun...@scipy.org>
<mailto:numpy-discussion-boun...@scipy.org
<mailto:numpy-discussion-boun...@scipy.org>>
<mailto:numpy-discussion-boun...@scipy.org
<mailto:numpy-discussion-boun...@scipy.org>
<mailto:numpy-discussion-boun...@scipy.org
<mailto:numpy-discussion-boun...@scipy.org>>>] *On Behalf Of *Todd
*Sent:* Wednesday, October 26, 2016 4:04 PM
*To:* Discussion of Numerical Python
<numpy-discussion@scipy.org
<mailto:numpy-discussion@scipy.org>
<mailto:numpy-discussion@scipy.org
<mailto:numpy-discussion@scipy.org>>
<mailto:numpy-discussion@scipy.org
<mailto:numpy-discussion@scipy.org>
<mailto:numpy-discussion@scipy.org
<mailto:numpy-discussion@scipy.org>>>>
*Subject:* Re: [Numpy-discussion] Intel random
number
package




On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr
<oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>
<mailto:oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>>
<mailto:oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>

<mailto:oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>>>>
wrote:

Another point already raised by Nathaniel is
that for
numpy's randomness ideally should provide a
way to
override
default algorithm for sampling from a particular
distribution.  For example RandomState
object that
implements PCG may rely o

Re: [Numpy-discussion] Intel random number package

[Julian Taylor]

On 10/27/2016 04:52 PM, Todd wrote:

On Thu, Oct 27, 2016 at 10:43 AM, Julian Taylor
<jtaylor.deb...@googlemail.com <mailto:jtaylor.deb...@googlemail.com>>
wrote:

On 10/27/2016 04:30 PM, Todd wrote:

On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers
<ralf.gomm...@gmail.com <mailto:ralf.gomm...@gmail.com>
<mailto:ralf.gomm...@gmail.com <mailto:ralf.gomm...@gmail.com>>>
wrote:


On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr
<oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>
<mailto:oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>>> wrote:

Please see responses inline.



*From:*NumPy-Discussion
[mailto:numpy-discussion-boun...@scipy.org
<mailto:numpy-discussion-boun...@scipy.org>
<mailto:numpy-discussion-boun...@scipy.org
<mailto:numpy-discussion-boun...@scipy.org>>] *On Behalf Of *Todd
*Sent:* Wednesday, October 26, 2016 4:04 PM
*To:* Discussion of Numerical Python
<numpy-discussion@scipy.org <mailto:numpy-discussion@scipy.org>
<mailto:numpy-discussion@scipy.org
<mailto:numpy-discussion@scipy.org>>>
*Subject:* Re: [Numpy-discussion] Intel random number
package




On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr
<oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>
<mailto:oleksandr.pav...@intel.com
<mailto:oleksandr.pav...@intel.com>>>
wrote:

Another point already raised by Nathaniel is that for
numpy's randomness ideally should provide a way to
override
default algorithm for sampling from a particular
distribution.  For example RandomState object that
implements PCG may rely on default acceptance-rejection
algorithm for sampling from Gamma, while the RandomState
object that provides interface to MKL might want to call
into MKL directly.



The approach that pyfftw uses at least for scipy, which
may also
work here, is that you can monkey-patch the
scipy.fftpack module
at runtime, replacing it with pyfftw's drop-in replacement.
scipy then proceeds to use pyfftw instead of its built-in
fftpack implementation.  Might such an approach work here?
Users can either use this alternative randomstate
replacement
directly, or they can replace numpy's with it at runtime and
numpy will then proceed to use the alternative.


The only reason that pyfftw uses monkeypatching is that the
better
approach is not possible due to license constraints with
FFTW (it's
GPL).


Yes, that is exactly why I brought it up.  Better approaches are
also
not possible with MKL due to license constraints.  It is a very
similar
situation overall.


Its not that similar, the better approach is certainly possible with
FFTW, the GPL is compatible with numpys license. It is only a
concern users of binary distributions. Nobody provided the code to
use fftw yet, but it would certainly be accepted.


Although it is technically compatible, it would make numpy effectively
GPL.  Suggestions for this have been explicitly rejected on these
grounds [1]

[1] https://github.com/numpy/numpy/issues/3485



Yes it would make numpy GPL, but that is not a concern for a lot of 
users. Users for who it is a problem can still use the non-GPL version.
A more interesting debate is whether our binary wheels should then be 
GPL wheels by default or not. Probably not, but that is something that 
should be discussed when its an actual issue.


But to clarify what I said, it would be accepted if the value it 
provides is sufficient compared to the code maintenance it adds. Given 
that pyfftw already exists the value is probably relatively small, but 
personally I'd still be interested in code that allows switching the fft 
backend as that could also allow plugging e.g. gpu based implementations 
(though again this is already covered by other third party modules).

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Intel random number package

[Julian Taylor]

On 10/27/2016 04:30 PM, Todd wrote:

On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers > wrote:


On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr
> wrote:

Please see responses inline.



*From:*NumPy-Discussion
[mailto:numpy-discussion-boun...@scipy.org
] *On Behalf Of *Todd
*Sent:* Wednesday, October 26, 2016 4:04 PM
*To:* Discussion of Numerical Python >
*Subject:* Re: [Numpy-discussion] Intel random number package




On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr
>
wrote:

Another point already raised by Nathaniel is that for
numpy's randomness ideally should provide a way to override
default algorithm for sampling from a particular
distribution.  For example RandomState object that
implements PCG may rely on default acceptance-rejection
algorithm for sampling from Gamma, while the RandomState
object that provides interface to MKL might want to call
into MKL directly.



The approach that pyfftw uses at least for scipy, which may also
work here, is that you can monkey-patch the scipy.fftpack module
at runtime, replacing it with pyfftw's drop-in replacement.
scipy then proceeds to use pyfftw instead of its built-in
fftpack implementation.  Might such an approach work here?
Users can either use this alternative randomstate replacement
directly, or they can replace numpy's with it at runtime and
numpy will then proceed to use the alternative.


The only reason that pyfftw uses monkeypatching is that the better
approach is not possible due to license constraints with FFTW (it's
GPL).


Yes, that is exactly why I brought it up.  Better approaches are also
not possible with MKL due to license constraints.  It is a very similar
situation overall.



Its not that similar, the better approach is certainly possible with 
FFTW, the GPL is compatible with numpys license. It is only a concern 
users of binary distributions. Nobody provided the code to use fftw yet, 
but it would certainly be accepted.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Intel random number package

[Julian Taylor]

On 10/26/2016 06:00 PM, Julian Taylor wrote:

On 10/26/2016 10:59 AM, Ralf Gommers wrote:



On Wed, Oct 26, 2016 at 8:33 PM, Julian Taylor
<jtaylor.deb...@googlemail.com <mailto:jtaylor.deb...@googlemail.com>>
wrote:

On 26.10.2016 06:34, Charles R Harris wrote:
> Hi All,
>
> There is a proposed random number package PR now up on github:
> https://github.com/numpy/numpy/pull/8209
<https://github.com/numpy/numpy/pull/8209>. It is from
> oleksandr-pavlyk <https://github.com/oleksandr-pavlyk
<https://github.com/oleksandr-pavlyk>> and implements
> the number random number package using MKL for increased speed.
I think
> we are definitely interested in the improved speed, but I'm not
sure
> numpy is the best place to put the package. I'd welcome any
comments on
> the PR itself, as well as any thoughts on the best way organize
or use
> of this work. Maybe scikit-random


Note that this thread is a continuation of
https://mail.scipy.org/pipermail/numpy-discussion/2016-July/075822.html



I'm not a fan of putting code depending on a proprietary library
into numpy.
This should be a standalone package which may provide the same
interface
as numpy.


I don't really see a problem with that in principle. Numpy can use Intel
MKL (and Accelerate) as well if it's available. It needs some thought
put into the API though - a ``numpy.random_intel`` module is certainly
not what we want.



For me there is a difference between being able to optionally use a
proprietary library as an alternative to free software libraries if the
user wishes to do so and offering functionality that only works with
non-free software.
We are providing a form of advertisement for them by allowing it (hey if
you buy this black box that you cannot modify or use freely you get this
neat numpy feature!).

I prefer for the full functionality of numpy to stay available with a
stack of community owned software, even if it may be less powerful that
way.


But then if this is really just the same random numbers numpy already 
provides just faster, it is probably acceptable in principle. I haven't 
actually looked at the PR yet.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Intel random number package

[Julian Taylor]

On 10/26/2016 10:59 AM, Ralf Gommers wrote:



On Wed, Oct 26, 2016 at 8:33 PM, Julian Taylor
<jtaylor.deb...@googlemail.com <mailto:jtaylor.deb...@googlemail.com>>
wrote:

On 26.10.2016 06:34, Charles R Harris wrote:
> Hi All,
>
> There is a proposed random number package PR now up on github:
> https://github.com/numpy/numpy/pull/8209
<https://github.com/numpy/numpy/pull/8209>. It is from
> oleksandr-pavlyk <https://github.com/oleksandr-pavlyk
<https://github.com/oleksandr-pavlyk>> and implements
> the number random number package using MKL for increased speed. I think
> we are definitely interested in the improved speed, but I'm not sure
> numpy is the best place to put the package. I'd welcome any comments on
> the PR itself, as well as any thoughts on the best way organize or use
> of this work. Maybe scikit-random


Note that this thread is a continuation of
https://mail.scipy.org/pipermail/numpy-discussion/2016-July/075822.html



I'm not a fan of putting code depending on a proprietary library
into numpy.
This should be a standalone package which may provide the same interface
as numpy.


I don't really see a problem with that in principle. Numpy can use Intel
MKL (and Accelerate) as well if it's available. It needs some thought
put into the API though - a ``numpy.random_intel`` module is certainly
not what we want.



For me there is a difference between being able to optionally use a 
proprietary library as an alternative to free software libraries if the 
user wishes to do so and offering functionality that only works with 
non-free software.
We are providing a form of advertisement for them by allowing it (hey if 
you buy this black box that you cannot modify or use freely you get this 
neat numpy feature!).


I prefer for the full functionality of numpy to stay available with a 
stack of community owned software, even if it may be less powerful that way.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Intel random number package

[Julian Taylor]

On 26.10.2016 06:34, Charles R Harris wrote:
> Hi All,
> 
> There is a proposed random number package PR now up on github:
> https://github.com/numpy/numpy/pull/8209. It is from
> oleksandr-pavlyk  and implements
> the number random number package using MKL for increased speed. I think
> we are definitely interested in the improved speed, but I'm not sure
> numpy is the best place to put the package. I'd welcome any comments on
> the PR itself, as well as any thoughts on the best way organize or use
> of this work. Maybe scikit-random
> 

I'm not a fan of putting code depending on a proprietary library into numpy.
This should be a standalone package which may provide the same interface
as numpy.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] automatically avoiding temporary arrays

[Julian Taylor]

On 03.10.2016 20:23, Chris Barker wrote:
> 
> 
> On Mon, Oct 3, 2016 at 3:16 AM, Julian Taylor
> <jtaylor.deb...@googlemail.com <mailto:jtaylor.deb...@googlemail.com>>
> wrote:
> 
> the problem with this approach is that we don't really want numpy
> hogging on to hundreds of megabytes of memory by default so it would
> need to be a user option.
> 
> 
> indeed -- but one could set an LRU cache to be very small (few items,
> not small memory), and then it get used within expressions, but not hold
> on to much outside of expressions.

numpy doesn't see the whole expression so we can't really do much.
(technically we could in 3.5 by using pep 523, but that would be a
larger undertaking)

> 
> However, is the allocation the only (Or even biggest) source of the
> performance hit?
>  

on large arrays the allocation is insignificant. What does cost some
time is faulting the memory into the process which implies writing zeros
into the pages (a page at a time as it is being used).
By storing memory blocks in numpy we would save this portion. This is
really the job of the libc, but these are usually tuned for general
purpose workloads and thus tend to give back memory back to the system
much earlier than numerical workloads would like.

Note that numpy already has a small memory block cache but its only used
for very small arrays where the allocation cost itself is significant,
it is limited to a couple megabytes at most.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] automatically avoiding temporary arrays

[Julian Taylor]

On 30.09.2016 23:09, josef.p...@gmail.com wrote:
> On Fri, Sep 30, 2016 at 9:38 AM, Julian Taylor
> <jtaylor.deb...@googlemail.com> wrote:
>> hi,
>> Temporary arrays generated in expressions are expensive as the imply
>> extra memory bandwidth which is the bottleneck in most numpy operations.
>> For example:
>>
>> r = a + b + c
>>
>> creates the b + c temporary and then adds a to it.
>> This can be rewritten to be more efficient using inplace operations:
>>
>> r = b + c
>> r += a
> 
> general question (I wouldn't understand the details even if I looked.)
> 
> how is this affected by broadcasting and type promotion?
> 
> Some of the main reasons that I don't like to use inplace operation in
> general is that I'm often not sure when type promotion occurs and when
> arrays expand during broadcasting.
> 
> for example b + c is 1-D, a is 2-D, and r has the broadcasted shape.
> another case when I switch away from broadcasting is when b + c is int
> or bool and a is float. Thankfully, we get error messages for casting
> now.

the temporary is only avoided when the casting follows the safe rule, so
it should be the same as what you get without inplace operations. E.g.
float32-temporary + float64 will not be converted to the unsafe float32
+= float64 which a normal inplace operations would allow. But
float64-temp + float32 is transformed.

Currently the only broadcasting that will be transformed is temporary +
scalar value, otherwise it will only work on matching array sizes.
Though there is not really anything that prevents full broadcasting but
its not implemented yet in the PR.

> 
>>
>> This saves some memory bandwidth and can speedup the operation by 50%
>> for very large arrays or even more if the inplace operation allows it to
>> be completed completely in the cpu cache.
> 
> I didn't realize the difference can be so large. That would make
> streamlining some code worth the effort.
> 
> Josef
> 
> 
>>
>> The problem is that inplace operations are a lot less readable so they
>> are often only used in well optimized code. But due to pythons
>> refcounting semantics we can actually do some inplace conversions
>> transparently.
>> If an operand in python has a reference count of one it must be a
>> temporary so we can use it as the destination array. CPython itself does
>> this optimization for string concatenations.
>>
>> In numpy we have the issue that we can be called from the C-API directly
>> where the reference count may be one for other reasons.
>> To solve this we can check the backtrace until the python frame
>> evaluation function. If there are only numpy and python functions in
>> between that and our entry point we should be able to elide the temporary.
>>
>> This PR implements this:
>> https://github.com/numpy/numpy/pull/7997
>>
>> It currently only supports Linux with glibc (which has reliable
>> backtraces via unwinding) and maybe MacOS depending on how good their
>> backtrace is. On windows the backtrace APIs are different and I don't
>> know them but in theory it could also be done there.
>>
>> A problem is that checking the backtrace is quite expensive, so should
>> only be enabled when the involved arrays are large enough for it to be
>> worthwhile. In my testing this seems to be around 180-300KiB sized
>> arrays, basically where they start spilling out of the CPU L2 cache.
>>
>> I made a little crappy benchmark script to test this cutoff in this branch:
>> https://github.com/juliantaylor/numpy/tree/elide-bench
>>
>> If you are interested you can run it with:
>> python setup.py build_ext -j 4 --inplace
>> ipython --profile=null check.ipy
>>
>> At the end it will plot the ratio between elided and non-elided runtime.
>> It should get larger than one around 180KiB on most cpus.
>>
>> If no one points out some flaw in the approach, I'm hoping to get this
>> into the next numpy version.
>>
>> cheers,
>> Julian
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Using library-specific headers

[Julian Taylor]

On 09/27/2016 11:09 PM, Pavlyk, Oleksandr wrote:

Suppose I would like to take advantage of some functions from MKL in
numpy C source code, which would require to use



#include “mkl.h”



Ideally this include line must not break the build of numpy when MKL is
not present, so my initial approach was to use



#if defined(SCIPY_MKL_H)

#include “mkl.h”

#endif



Unfortunately, this did not work when building with gcc on a machine
where MKL is present on default LD_LIBRARY_PATH, because then the
distutils code was setting SCIPY_MKL_H preprocessor variable, even
though mkl headers are not on the C_INCLUDE_PATH.



What is the preferred solution to include an external library header to
ensure that code-base continues to build in most common cases?



One approach I can think of is to set a preprocessor variable, say
HAVE_MKL_HEADERS in numpy/core/includes/numpy/config.h depending on an
outcome of building of a simple _configtest.c using
config.try_compile(), like it is done in numpy/core/setup.py //

/ /

Is there a simpler, or a better way?



hi,
you could put the header into OPTIONAL_HEADERS in 
numpy/core/setup_common.py. This will define HAVE_HEADERFILENAME_H for 
you but this will not check that the corresponding the library actually 
exists and can be linked.
For that SCIPY_MKL_H is probably the right macro, though its name is 
confusing as it does not check for the header presence ...


Can you tell us more about what from mkl you are attempting to add and 
for what purpos, e.g. is it something that should go into numpy proper 
or just for personal/internal use?


cheers,
Julian
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy threads crash when allocating arrays

[Julian Taylor]

On 14.06.2016 19:34, Burlen Loring wrote:



here's my question: given Py_BEGIN_ALLOW_THREADS is used by numpy how
can numpy be thread safe? and how can someone using the C-API know where
it's necessary to acquire the GIL? Maybe someone can explain this?



numpy only releases the GIL when it is not accessing any python objects 
or other non-threadsafe structures anymore.

That is usually during computation loops and IO.


Your problem is indeed a missing PyGILState_Ensure

I am assuming that the threads you are using are not created by python, 
so you don't have a threadstate setup and no GIL.
You do set it up with that function, see 
https://docs.python.org/2/c-api/init.html#non-python-created-threads

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Calling C code that assumes SIMD aligned data.

[Julian Taylor]

note that anything larger than 16 bytes alignment is unnecessary for 
simd purposes on current hardware (>= haswell). 16 byte is default 
malloc alignment on amd64.

And even on older ones (sandy bridge) the penalty is pretty minor.

On 05.05.2016 22:32, Charles R Harris wrote:



On Thu, May 5, 2016 at 2:10 PM, Øystein Schønning-Johansen
> wrote:

Thanks for your answer, Francesc. Knowing that there is no numpy
solution saves the work of searching for this. I've not tried the
solution described at SO, but it looks like a real performance
killer. I'll rather try to override malloc with glibs malloc_hooks
or LD_PRELOAD tricks. Do you think that will do it? I'll try it and
report back.

Thanks,
-Øystein


Might take a look at how numpy handles this in
`numpy/core/src/umath/simd.inc.src`.



Chuck


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0rc1 released.

[Julian Taylor]

that test needs about 500Mb of memory on windows as it doesn't have
sparse allocations like most *nixes.
It used to fail for me during release testing when I only gave the
windows VM 1GB of ram.
If its a problem for CI we could disable it on windows, or at least skip
the complex double case.

On 23.02.2016 21:40, Charles R Harris wrote:
> Christoph reports the following problem that I am unable to reproduce on
> appveyor or find reported elsewhere.
> 
> On all 32-bit platforms:
> 
> 
> ERROR: test_zeros_big (test_multiarray.TestCreation)
> 
> Traceback (most recent call last):
>   File
> "X:\Python27\lib\site-packages\numpy\core\tests\test_multiarray.py",
> line 594, in test_zeros_big
> d = np.zeros((30 * 1024**2,), dtype=dt)
> MemoryError
> 
> I would be much obliged if someone else could demonstrate it.
> 
> 
> 
> Chuck
> 
> 
> 
> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fwd: Multi-distribution Linux wheels - please test

[Julian Taylor]

On 09.02.2016 04:59, Nathaniel Smith wrote:
> On Mon, Feb 8, 2016 at 6:07 PM, Nathaniel Smith  wrote:
>> On Mon, Feb 8, 2016 at 6:04 PM, Matthew Brett  
>> wrote:
>>> On Mon, Feb 8, 2016 at 5:26 PM, Nathaniel Smith  wrote:
 On Mon, Feb 8, 2016 at 4:37 PM, Matthew Brett  
 wrote:
 [...]
> I can't replicate the segfault with manylinux wheels and scipy.  On
> the other hand, I get a new test error for numpy from manylinux, scipy
> from manylinux, like this:
>
> $ python -c 'import scipy.linalg; scipy.linalg.test()'
>
> ==
> FAIL: test_decomp.test_eigh('general ', 6, 'F', True, False, False, (2, 
> 4))
> --
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line
> 197, in runTest
> self.test(*self.arg)
>   File 
> "/usr/local/lib/python2.7/dist-packages/scipy/linalg/tests/test_decomp.py",
> line 658, in eigenhproblem_general
> assert_array_almost_equal(diag2_, ones(diag2_.shape[0]), 
> DIGITS[dtype])
>   File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py",
> line 892, in assert_array_almost_equal
> precision=decimal)
>   File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py",
> line 713, in assert_array_compare
> raise AssertionError(msg)
> AssertionError:
> Arrays are not almost equal to 4 decimals
>
> (mismatch 100.0%)
>  x: array([ 0.,  0.,  0.], dtype=float32)
>  y: array([ 1.,  1.,  1.])
>
> --
> Ran 1507 tests in 14.928s
>
> FAILED (KNOWNFAIL=4, SKIP=1, failures=1)
>
> This is a very odd error, which we don't get when running over a numpy
> installed from source, linked to ATLAS, and doesn't happen when
> running the tests via:
>
> nosetests /usr/local/lib/python2.7/dist-packages/scipy/linalg
>
> So, something about the copy of numpy (linked to openblas) is
> affecting the results of scipy (also linked to openblas), and only
> with a particular environment / test order.
>
> If you'd like to try and see whether y'all can do a better job of
> debugging than me:
>
> # Run this script inside a docker container started with this incantation:
> # docker run -ti --rm ubuntu:12.04 /bin/bash
> apt-get update
> apt-get install -y python curl
> apt-get install libpython2.7  # this won't be necessary with next
> iteration of manylinux wheel builds
> curl -LO https://bootstrap.pypa.io/get-pip.py
> python get-pip.py
> pip install -f https://nipy.bic.berkeley.edu/manylinux numpy scipy nose
> python -c 'import scipy.linalg; scipy.linalg.test()'

 I just tried this and on my laptop it completed without error.

 Best guess is that we're dealing with some memory corruption bug
 inside openblas, so it's getting perturbed by things like exactly what
 other calls to openblas have happened (which is different depending on
 whether numpy is linked to openblas), and which core type openblas has
 detected.

 On my laptop, which *doesn't* show the problem, running with
 OPENBLAS_VERBOSE=2 says "Core: Haswell".

 Guess the next step is checking what core type the failing machines
 use, and running valgrind... anyone have a good valgrind suppressions
 file?
>>>
>>> My machine (which does give the failure) gives
>>>
>>> Core: Core2
>>>
>>> with OPENBLAS_VERBOSE=2
>>
>> Yep, that allows me to reproduce it:
>>
>> root@f7153f0cc841:/# OPENBLAS_VERBOSE=2 OPENBLAS_CORETYPE=Core2 python
>> -c 'import scipy.linalg; scipy.linalg.test()'
>> Core: Core2
>> [...]
>> ==
>> FAIL: test_decomp.test_eigh('general ', 6, 'F', True, False, False, (2, 4))
>> --
>> [...]
>>
>> So this is indeed sounding like an OpenBLAS issue... next stop
>> valgrind, I guess :-/
> 
> Here's the valgrind output:
>   https://gist.github.com/njsmith/577d028e79f0a80d2797
> 
> There's a lot of it, but no smoking guns have jumped out at me :-/
> 
> -n
> 

plenty of smoking guns, e.g.:

.==3695== Invalid read of size 8
3417==3695==at 0x7AAA9C0: daxpy_k_CORE2 (in
/usr/local/lib/python2.7/dist-packages/numpy/.libs/libopenblas.so.0)
3418==3695==by 0x76BEEFC: ger_kernel (in
/usr/local/lib/python2.7/dist-packages/numpy/.libs/libopenblas.so.0)
3419==3695==by 0x788F618: exec_blas (in
/usr/local/lib/python2.7/dist-packages/numpy/.libs/libopenblas.so.0)
3420==3695==by 0x76BF099: dger_thread (in

Re: [Numpy-discussion] Fwd: Multi-distribution Linux wheels - please test

[Julian Taylor]

On 09.02.2016 21:01, Nathaniel Smith wrote:
> On Tue, Feb 9, 2016 at 11:37 AM, Julian Taylor
> <jtaylor.deb...@googlemail.com> wrote:
>> On 09.02.2016 04:59, Nathaniel Smith wrote:
>>> On Mon, Feb 8, 2016 at 6:07 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>>> On Mon, Feb 8, 2016 at 6:04 PM, Matthew Brett <matthew.br...@gmail.com> 
>>>> wrote:
>>>>> On Mon, Feb 8, 2016 at 5:26 PM, Nathaniel Smith <n...@pobox.com> wrote:
>>>>>> On Mon, Feb 8, 2016 at 4:37 PM, Matthew Brett <matthew.br...@gmail.com> 
>>>>>> wrote:
>>>>>> [...]
>>>>>>> I can't replicate the segfault with manylinux wheels and scipy.  On
>>>>>>> the other hand, I get a new test error for numpy from manylinux, scipy
>>>>>>> from manylinux, like this:
>>>>>>>
>>>>>>> $ python -c 'import scipy.linalg; scipy.linalg.test()'
>>>>>>>
>>>>>>> ==
>>>>>>> FAIL: test_decomp.test_eigh('general ', 6, 'F', True, False, False, (2, 
>>>>>>> 4))
>>>>>>> --
>>>>>>> Traceback (most recent call last):
>>>>>>>   File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line
>>>>>>> 197, in runTest
>>>>>>> self.test(*self.arg)
>>>>>>>   File 
>>>>>>> "/usr/local/lib/python2.7/dist-packages/scipy/linalg/tests/test_decomp.py",
>>>>>>> line 658, in eigenhproblem_general
>>>>>>> assert_array_almost_equal(diag2_, ones(diag2_.shape[0]), 
>>>>>>> DIGITS[dtype])
>>>>>>>   File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py",
>>>>>>> line 892, in assert_array_almost_equal
>>>>>>> precision=decimal)
>>>>>>>   File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py",
>>>>>>> line 713, in assert_array_compare
>>>>>>> raise AssertionError(msg)
>>>>>>> AssertionError:
>>>>>>> Arrays are not almost equal to 4 decimals
>>>>>>>
>>>>>>> (mismatch 100.0%)
>>>>>>>  x: array([ 0.,  0.,  0.], dtype=float32)
>>>>>>>  y: array([ 1.,  1.,  1.])
>>>>>>>
>>>>>>> --
>>>>>>> Ran 1507 tests in 14.928s
>>>>>>>
>>>>>>> FAILED (KNOWNFAIL=4, SKIP=1, failures=1)
>>>>>>>
>>>>>>> This is a very odd error, which we don't get when running over a numpy
>>>>>>> installed from source, linked to ATLAS, and doesn't happen when
>>>>>>> running the tests via:
>>>>>>>
>>>>>>> nosetests /usr/local/lib/python2.7/dist-packages/scipy/linalg
>>>>>>>
>>>>>>> So, something about the copy of numpy (linked to openblas) is
>>>>>>> affecting the results of scipy (also linked to openblas), and only
>>>>>>> with a particular environment / test order.
>>>>>>>
>>>>>>> If you'd like to try and see whether y'all can do a better job of
>>>>>>> debugging than me:
>>>>>>>
>>>>>>> # Run this script inside a docker container started with this 
>>>>>>> incantation:
>>>>>>> # docker run -ti --rm ubuntu:12.04 /bin/bash
>>>>>>> apt-get update
>>>>>>> apt-get install -y python curl
>>>>>>> apt-get install libpython2.7  # this won't be necessary with next
>>>>>>> iteration of manylinux wheel builds
>>>>>>> curl -LO https://bootstrap.pypa.io/get-pip.py
>>>>>>> python get-pip.py
>>>>>>> pip install -f https://nipy.bic.berkeley.edu/manylinux numpy scipy nose
>>>>>>> python -c 'import scipy.linalg; scipy.linalg.test()'
>>>>>>
>>>>>> I just tried this and on my laptop it completed without error.
>>>>>>
>>>>>> Best guess is that we're dealing with some memory corruption bug
>>>&

Re: [Numpy-discussion] Fwd: Multi-distribution Linux wheels - please test

[Julian Taylor]

On 09.02.2016 20:52, Matthew Brett wrote:
> On Mon, Feb 8, 2016 at 7:59 PM, Nathaniel Smith  wrote:
>> On Mon, Feb 8, 2016 at 6:07 PM, Nathaniel Smith  wrote:
>>> On Mon, Feb 8, 2016 at 6:04 PM, Matthew Brett  
>>> wrote:
 On Mon, Feb 8, 2016 at 5:26 PM, Nathaniel Smith  wrote:
> On Mon, Feb 8, 2016 at 4:37 PM, Matthew Brett  
> wrote:
> [...]
>> I can't replicate the segfault with manylinux wheels and scipy.  On
>> the other hand, I get a new test error for numpy from manylinux, scipy
>> from manylinux, like this:
>>
>> $ python -c 'import scipy.linalg; scipy.linalg.test()'
>>
>> ==
>> FAIL: test_decomp.test_eigh('general ', 6, 'F', True, False, False, (2, 
>> 4))
>> --
>> Traceback (most recent call last):
>>   File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line
>> 197, in runTest
>> self.test(*self.arg)
>>   File 
>> "/usr/local/lib/python2.7/dist-packages/scipy/linalg/tests/test_decomp.py",
>> line 658, in eigenhproblem_general
>> assert_array_almost_equal(diag2_, ones(diag2_.shape[0]), 
>> DIGITS[dtype])
>>   File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py",
>> line 892, in assert_array_almost_equal
>> precision=decimal)
>>   File "/usr/local/lib/python2.7/dist-packages/numpy/testing/utils.py",
>> line 713, in assert_array_compare
>> raise AssertionError(msg)
>> AssertionError:
>> Arrays are not almost equal to 4 decimals
>>
>> (mismatch 100.0%)
>>  x: array([ 0.,  0.,  0.], dtype=float32)
>>  y: array([ 1.,  1.,  1.])
>>
>> --
>> Ran 1507 tests in 14.928s
>>
>> FAILED (KNOWNFAIL=4, SKIP=1, failures=1)
>>
>> This is a very odd error, which we don't get when running over a numpy
>> installed from source, linked to ATLAS, and doesn't happen when
>> running the tests via:
>>
>> nosetests /usr/local/lib/python2.7/dist-packages/scipy/linalg
>>
>> So, something about the copy of numpy (linked to openblas) is
>> affecting the results of scipy (also linked to openblas), and only
>> with a particular environment / test order.
>>
>> If you'd like to try and see whether y'all can do a better job of
>> debugging than me:
>>
>> # Run this script inside a docker container started with this 
>> incantation:
>> # docker run -ti --rm ubuntu:12.04 /bin/bash
>> apt-get update
>> apt-get install -y python curl
>> apt-get install libpython2.7  # this won't be necessary with next
>> iteration of manylinux wheel builds
>> curl -LO https://bootstrap.pypa.io/get-pip.py
>> python get-pip.py
>> pip install -f https://nipy.bic.berkeley.edu/manylinux numpy scipy nose
>> python -c 'import scipy.linalg; scipy.linalg.test()'
>
> I just tried this and on my laptop it completed without error.
>
> Best guess is that we're dealing with some memory corruption bug
> inside openblas, so it's getting perturbed by things like exactly what
> other calls to openblas have happened (which is different depending on
> whether numpy is linked to openblas), and which core type openblas has
> detected.
>
> On my laptop, which *doesn't* show the problem, running with
> OPENBLAS_VERBOSE=2 says "Core: Haswell".
>
> Guess the next step is checking what core type the failing machines
> use, and running valgrind... anyone have a good valgrind suppressions
> file?

 My machine (which does give the failure) gives

 Core: Core2

 with OPENBLAS_VERBOSE=2
>>>
>>> Yep, that allows me to reproduce it:
>>>
>>> root@f7153f0cc841:/# OPENBLAS_VERBOSE=2 OPENBLAS_CORETYPE=Core2 python
>>> -c 'import scipy.linalg; scipy.linalg.test()'
>>> Core: Core2
>>> [...]
>>> ==
>>> FAIL: test_decomp.test_eigh('general ', 6, 'F', True, False, False, (2, 4))
>>> --
>>> [...]
>>>
>>> So this is indeed sounding like an OpenBLAS issue... next stop
>>> valgrind, I guess :-/
>>
>> Here's the valgrind output:
>>   https://gist.github.com/njsmith/577d028e79f0a80d2797
>>
>> There's a lot of it, but no smoking guns have jumped out at me :-/
> 
> Could you send me instructions on replicating the valgrind run, I'll
> run on on the actual Core2 machine...
> 
> Matthew


please also use this suppression file, should reduce the python noise
significantly but it might be a bit out of date. Used to work fine on an
ubuntu built python.
#
# This is a valgrind suppression file that 

Re: [Numpy-discussion] Multi-distribution Linux wheels - please test

[Julian Taylor]

On 02/08/2016 05:23 PM, Daπid wrote:
> 
> On 8 February 2016 at 16:19, Olivier Grisel  > wrote:
> 
> 
> 
> OPENBLAS_CORETYPE=Nehalem python3 -c "import numpy as np; from scipy
> import linalg; linalg.eigh(np.random.randn(200, 200))"
> 
> So this is an issue with the architecture detection of OpenBLAS.
> 
> 
> I am seeing the same problem on a native Linux box, with Ivy Bridge
> processor (i5-3317U). According to your script, both my native openblas
> and the one in the wheel recognises my CPU as Sandybridge, but the wheel
> produces a segmentation fault. Setting the architecture to Nehalem works.
> 

more likely that is a bug the kernel of openblas instead of its cpu
detection.
The cpuinfo of Oliver indicates its at least a sandy bridge, and ivy
bridge is be sandy bridge compatible.
Is an up to date version of openblas used?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Linking other libm-Implementation

[Julian Taylor]

On 02/08/2016 06:36 PM, Nathaniel Smith wrote:
> On Feb 8, 2016 3:04 AM, "Nils Becker"  > wrote:
>>
> [...]
>> Very superficial benchmarks (see below) seem devastating for gnu libm.
> It seems that openlibm (compiled with gcc -mtune=native -O3) performs
> really well and intels libm implementation is the best (on my intel
> CPU). I did not check the accuracy of the functions, though.
>>
>> My own code uses a lot of trigonometric and complex functions (optics
> calculations). I'd guess it could go 25% faster by just using a better
> libm implementation. Therefore, I have an interest in getting sane
> linking to a defined libm implementation to work.
> 
> On further thought: I guess that to do this we actually will need to
> change the names of the functions in openlibm and then use those names
> when calling from numpy. So long as we're using the regular libm symbol
> names, it doesn't matter what library the python extensions themselves
> are linked to; the way ELF symbol lookup works, the libm that the python
> interpreter is linked to will be checked *before* checking the libm that
> numpy is linked to, so the symbols will all get shadowed.
> 
> I guess statically linking openlibm would also work, but not sure that's
> a great idea since we'd need it multiple places.
> 
>> Apparently openlibm seems quite a good choice for numpy, at least
> performance wise. However, I did not find any documentation or tests of
> the accuracy of its functions. A benchmarking and testing (for accuracy)
> code for libms would probably be a good starting point for a discussion.
> I could maybe help with that - but apparently not with any
> linking/building stuff (I just don't get it).
>>
>> Benchmark:
>>
>> gnu libm.so
>> 3000 x sin(double[10]):  6.68215647800389 s
>> 3000 x log(double[10]):  8.86350397899514 s
>> 3000 x exp(double[10]):  6.560557693999726 s
>>
>> openlibm.so
>> 3000 x sin(double[10]):  4.5058218560006935 s
>> 3000 x log(double[10]):  4.106520485998772 s
>> 3000 x exp(double[10]):  4.597905882001214 s
>>
>> Intel libimf.so
>> 3000 x sin(double[10]):  4.282402812998043 s
>> 3000 x log(double[10]):  4.008453270995233 s
>> 3000 x exp(double[10]):  3.30127963848 s
> 
> I would be highly suspicious that this speed comes at the expense of
> accuracy... My impression is that there's a lot of room to make
> speed/accuracy tradeoffs in these functions, and modern glibc's libm has
> seen a fair amount of scrutiny by people who have access to the same
> code that openlibm is based off of. But then again, maybe not :-).
> 
> If these are the operations that you care about optimizing, an even
> better approach might be to figure out how to integrate a vector math
> library here like yeppp (BSD licensed) or MKL. Libm tries to optimize
> log(scalar); these are libraries that specifically try to optimize
> log(vector). Adding this would require changing numpy's code to use
> these new APIs though. (Very new gcc can also try to do this in some
> cases but I don't know how good at it it is... Julian might.)
> 
> -n


which version of glibm was used here? There are significant difference
in performance between versions.
Also the input ranges are very important for these functions, depending
on input the speed of these functions can vary by factors of 1000.

glibm now includes vectorized versions of most math functions, does
openlibm have vectorized math?
Thats where most speed can be gained, a lot more than 25%.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy pull requests getting out of hand.

[Julian Taylor]

I don't like that approach, closing PRs with valuable code causes them
to get lost in the much larger set of closed ones.
Instead we could tag them appropriately so they can be found by
parties interested in using or finishing them. These could also be used
for new contributors to work on.
You could also tag and close them, but I find it makes them harder to
discover especially for outsiders who are not aware of this policy.

Also we could extend our contribution guidelines to note that
reviewing existing PRs can be a much more valuable contribution than
adding new code. Possibly adding some review guidelines.


On 31.01.2016 20:25, Jeff Reback wrote:
> FYI also useful to simply close by time - say older than 6 months with a 
> message for the writer to reopen if they want to work on it
> 
> then u don't get too many stale ones 
> 
> my 2c
> 
>> On Jan 31, 2016, at 2:10 PM, Charles R Harris  
>> wrote:
>>
>> Hi All,
>>
>> There are now 130 open numpy pull requests and it seems almost impossible to 
>> keep that number down. My personal decision is that I am going to ignore any 
>> new enhancements for the next couple of months and only merge bug fixes, 
>> tests, house keeping (style, docs, deprecations), and old PRs. I would also 
>> request that other maintainers start looking a taking care of older PRs, 
>> either cleaning them up and merging, or closing them.
>>
>> Chuck
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Julian Taylor]

hi,
even if it are good changes, I find it reasonable to ask for a delay in
numpy release if you need more time to adapt. Of course this has to be
within reason and can be rejected, but its very valuable to know changes
break existing old workarounds. If pyfits broke there is probably a lot
more code we don't know about that is also broken.

Sometimes we might even be able to get the good without breaking the
bad. E.g. thanks to Sebastians heroic efforts in his recent indexing
rewrite only very little broke and a lot of odd stuff could be equipped
with deprecation warnings instead of breaking.
Of course that cannot often be done or be worthwhile but its at least
worth considering when we change core functionality.

cheers,
Julian


On 31.01.2016 22:52, Marten van Kerkwijk wrote:
> Hi Julian,
> 
> While the numpy 1.10 situation was bad, I do want to clarify that the
> problems we had in astropy were a consequence of *good* changes in
> `recarray`, which solved many problems, but also broke the work-arounds
> that had been created in `astropy.io.fits` quite a long time ago
> (possibly before astropy became as good as it tries to be now at moving
> issues upstream and perhaps before numpy had become as responsive to
> what happens downstream as it is now; I think it is fair to say many
> project's attitude to testing has changed rather drastically in the last
> decade!).
> 
> I do agree, though, that it just goes to show one has to try to be
> careful, and like Nathaniel's suggestion of automatic testing with
> pre-releases -- I just asked on our astropy-dev list whether we can
> implement it.
> 
> All the best,
> 
> Marten
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Julian Taylor]

On 01/30/2016 06:27 PM, Ralf Gommers wrote:
> 
> 
> On Fri, Jan 29, 2016 at 11:39 PM, Nathaniel Smith  > wrote:
> 
> It occurs to me that the best solution might be to put together a
> .travis.yml for the release branches that does: "for pkg in
> IMPORTANT_PACKAGES: pip install $pkg; python -c 'import pkg;
> pkg.test()'"
> This might not be viable right now, but will be made more viable if
> pypi starts allowing official Linux wheels, which looks likely to
> happen before 1.12... (see PEP 513)
> 
> On Jan 29, 2016 9:46 AM, "Andreas Mueller"  > wrote:
> >
> > Is this the point when scikit-learn should build against it?
> 
> Yes please!
> 
> > Or do we wait for an RC?
> 
> This is still all in flux, but I think we might actually want a rule
> that says it can't become an RC until after we've tested
> scikit-learn (and a list of similarly prominent packages). On the
> theory that RC means "we think this is actually good enough to
> release" :-). OTOH I'm not sure the alpha/beta/RC distinction is
> very helpful; maybe they should all just be betas.
> 
> > Also, we need a scipy build against it. Who does that?
> 
> Like Julian says, it shouldn't be necessary. In fact using old
> builds of scipy and scikit-learn is even better than rebuilding
> them, because it tests numpy's ABI compatibility -- if you find you
> *have* to rebuild something then we *definitely* want to know that.
> 
> > Our continuous integration doesn't usually build scipy or numpy, so it 
> will be a bit tricky to add to our config.
> > Would you run our master tests? [did we ever finish this discussion?]
> 
> We didn't, and probably should... :-)
> 
> Why would that be necessary if scikit-learn simply tests pre-releases of
> numpy as you suggested earlier in the thread (with --pre)?
> 
> There's also https://github.com/MacPython/scipy-stack-osx-testing by the
> way, which could have scikit-learn and scikit-image added to it.
> 
> That's two options that are imho both better than adding more workload
> for the numpy release manager. Also from a principled point of view,
> packages should test with new versions of their dependencies, not the
> other way around.


It would be nice but its not realistic, I doubt most upstreams that are
not themselves major downstreams are even subscribed to this list.

Testing or delegating testing of least our major downstreams should be
the job of the release manager.
Thus I also disagree with our more frequent releases. It puts too much
porting and testing effort on our downstreams and it gets infeaseble for
a volunteer release manager to handle.
I fear by doing this we will end up in an situation where more
downstreams put upper bounds on their supported numpy releases like e.g.
astropy already did.
This has bad consequences like the subclass breaking of linspace that
should have been caught month ago but was not because upstreams were
discouraging users from upgrading numpy because they could not keep up
with porting.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Julian Taylor]

You most likely don't need a scipy build against it. You should be able
to use the oldest scipy our project supports. Numpy does try to not
break its reverse dependencies, if stuff breaks it should only occur in
edge cases not affecting functionality of real applications (like
warnings or overzealous testing).

Of course that only works if people bother to test against the numpy
prereleases.


On 01/29/2016 06:45 PM, Andreas Mueller wrote:
> Is this the point when scikit-learn should build against it?
> Or do we wait for an RC?
> Also, we need a scipy build against it. Who does that?
> Our continuous integration doesn't usually build scipy or numpy, so it
> will be a bit tricky to add to our config.
> Would you run our master tests? [did we ever finish this discussion?]
> 
> Andy
> 
> On 01/28/2016 03:51 PM, Charles R Harris wrote:
>> Hi All,
>>
>> I hope I am pleased to announce the Numpy 1.11.0b2 release. The first
>> beta was a damp squib due to missing files in the released source
>> files, this release fixes that. The new source filese may be
>> downloaded from sourceforge, no binaries will be released until the
>> mingw tool chain problems are sorted.
>>
>> Please test and report any problem.
>>
>> Chuck
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[Julian Taylor]

On 09.01.2016 04:38, Nathaniel Smith wrote:
> On Fri, Jan 8, 2016 at 7:17 PM, Nathan Goldbaum  wrote:
>> Doesn't building on CentOS 5 also mean using a quite old version of gcc?
> 
> Yes. IIRC CentOS 5 ships with gcc 4.4, and you can bump that up to gcc
> 4.8 by using the Redhat Developer Toolset release (which is gcc +
> special backport libraries to let it generate RHEL5/CentOS5-compatible
> binaries). (I might have one or both of those version numbers slightly
> wrong.)
> 
>> I've never tested this, but I've seen claims on the anaconda mailing list of
>> ~25% slowdowns compared to building from source or using system packages,
>> which was attributed to building using an older gcc that doesn't optimize as
>> well as newer versions.
> 
> I'd be very surprised if that were a 25% slowdown in general, as
> opposed to a 25% slowdown on some particular inner loop that happened
> to neatly match some new feature in a new gcc (e.g. something where
> the new autovectorizer kicked in). But yeah, in general this is just
> an inevitable trade-off when it comes to distributing binaries: you're
> always going to pay some penalty for achieving broad compatibility as
> compared to artisanally hand-tuned binaries specialized for your
> machine's exact OS version, processor, etc. Not much to be done,
> really. At some point the baseline for compatibility will switch to
> "compile everything on CentOS 6", and that will be better but it will
> still be worse than binaries that target CentOS 7, and so on and so
> forth.
> 

I have over the years put in one gcc specific optimization after the
other so yes using an ancient version will make many parts significantly
slower. Though that is not really a problem, updating a compiler is easy
even without redhats devtoolset.

At least as far as numpy is concerned linux binaries should not be a
very big problem. The only dependency where the version matters is glibc
which has updated its interfaces we use (in a backward compatible way)
many times.
But here if we use a old enough baseline glibc (e.g. centos5 or ubuntu
10.04) we are fine at reasonable performance costs, basically only
slower memcpy.

Scipy on the other hand is a larger problem as it contains C++ code.
Linux systems are now transitioning to C++11 which is binary
incompatible in parts to the old standard. There a lot of testing is
necessary to check if we are affected.
How does Anaconda deal with C++11?



signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Defining a base linux-64 environment [was: Should I use pip install numpy in linux?]

[Julian Taylor]

On 09.01.2016 12:52, Robert McGibbon wrote:
> Hi all,
> 
> I went ahead and tried to collect a list of all of the libraries that
> could be considered to constitute the "base" system for linux-64. The
> strategy I used was to leverage off the work done by the folks at
> Continuum by searching through their pre-compiled binaries
> from https://repo.continuum.io/pkgs/free/linux-64/ to find shared
> libraries that were dependened on (according to ldd)  that were not
> accounted for by the declared dependencies that each package made known
> to the conda package manager.
> 

do those packages use ld --as-needed for linking?
there are a lot libraries in that list that I highly doubt are directly
used by the packages.



signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem writing array with savetxt (python3.5)

[Julian Taylor]

hi
unfortunately numpy text io in python3 is very broken and best avoided.
Technically you might be able to work around it by opening the file in
binary mode but that is the wrong way of doing it and might break when
we finally get around to really fixing it, also won't work for unicode
and string containing escape characters (e.g. windows filenames).


On 12/26/2015 12:47 PM, Maxim Mayzel wrote:
> Dear all,
> 
> I’m having a problem writing an array with np.savetxt, see below.
> (python3.5, numpy 1.10.1)
> 
> import numpy as np
> a=np.ones(5,dtype=int)
> np.savetxt('t.txt',a)
> 
> Works fine, but the content of ’t.txt’ is overwritten. 
> But, passing file handle gives an error, while it works fine on python2.7 !
> 
> 
> with open('t.txt','w') as t:
>np.savetxt(t,a,fmt='%i')
>   ...:
> ---
> TypeError Traceback (most recent call last)
> /Users/may/anaconda/lib/python3.5/site-packages/numpy/lib/npyio.py in 
> savetxt(fname, X, fmt, delimiter, newline, header, footer, comments)
>   1155 try:
> -> 1156 fh.write(asbytes(format % tuple(row) + newline))
>   1157 except TypeError:
> 
> TypeError: write() argument must be str, not bytes
> 
> During handling of the above exception, another exception occurred:
> 
> TypeError Traceback (most recent call last)
>  in ()
>  1 with open('t.txt','w') as t:
> > 2 np.savetxt(t,a,fmt='%i')
>  3
> 
> /Users/may/anaconda/lib/python3.5/site-packages/numpy/lib/npyio.py in 
> savetxt(fname, X, fmt, delimiter, newline, header, footer, comments)
>   1158 raise TypeError("Mismatch between array dtype 
> ('%s') and "
>   1159 "format specifier ('%s')"
> -> 1160 % (str(X.dtype), format))
>   1161 if len(footer) > 0:
>   1162 footer = footer.replace('\n', '\n' + comments)
> 
> TypeError: Mismatch between array dtype ('int64') and format specifier ('%i’)
> 
> 
> Best,
> Dr. Maxim Mayzel
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] When to stop supporting Python 2.6?

[Julian Taylor]

On 12/09/2015 12:10 AM, Ralf Gommers wrote:



On Wed, Dec 9, 2015 at 12:01 AM, Chris Barker > wrote:

drop 2.6

I still don't understand why folks insist that they need to run a
(very)) old python on an old OS, but need the latest and greatest numpy.

Chuck's list was pretty long and compelling.

-CHB



On Mon, Dec 7, 2015 at 1:38 AM, Sturla Molden
> wrote:

Charles R Harris > wrote:

> As a strawman proposal, how about dropping moving to 2.7 and 3.4 
minimum
> supported version next fall, say around numpy 1.12 or 1.13 depending 
on how
> the releases go.
>
> I would like to here from the scipy folks first.


+1 for dropping Python 2.6, 3.2 and 3.3 after branching 1.11.x. We're
already behind other projects like ipython, pandas and matplotlib as
usual, so there really isn't much point in being the only project
(together with scipy) of the core stack to keep on supporting more or
less obsolete Python versions.

Ralf



I don't see how that is a relevant point. NumPy is the lowest component 
of the stack, we have to be the last to drop support for Python 2.6. And 
we aren't yet the last even when only looking at the high profile 
components. Astropy still supports 2.6 for another release.
Though by the time 1.11 comes out we might be so I'm ok with dropping it 
after that even when I'm not convinced we gain anything significant from 
doing so.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] When to stop supporting Python 2.6?

[Julian Taylor]

dropping 3.2: +-0 as it would remove some extra code in our broken py3
string handling but not much
dropping 3.3: -1 doesn't gain us anything so far I know
dropping 2.6: -1, I don't see not enough advantage the only issue I know
of is an occasional set literal which gets caught by our test-suite
immediately. Besides 2.6 is still the default in RHEL6. But if there is
something larger which makes it worthwhile I don't know about I have no
objections.


On 04.12.2015 10:27, David Cournapeau wrote:
> I would be in favour of dropping 3.3, but not 2.6 until it becomes too
> cumbersome to support.
> 
> As a data point, as of april, 2.6 was more downloaded than all python
> 3.X versions together when looking at pypi
> numbers: https://caremad.io/2015/04/a-year-of-pypi-downloads/
> 
> David
> 
> On Thu, Dec 3, 2015 at 11:03 PM, Jeff Reback  > wrote:
> 
> pandas is going to drop
> 2.6 and 3.3 next release at end of Jan
> 
> (3.2 dropped in 0.17, in October)
> 
> 
> 
> I can be reached on my cell 917-971-6387
> > On Dec 3, 2015, at 6:00 PM, Bryan Van de Ven  > wrote:
> >
> >
> >> On Dec 3, 2015, at 4:59 PM, Eric Firing  > wrote:
> >>
> >> Chuck,
> >>
> >> I would support dropping the old versions now.  As a related data
> point, matplotlib is testing master on 2.7, 3.4, and 3.5--no more
> 2.6 and 3.3.
> >
> > Ditto for Bokeh.
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org 
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org 
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] msvc9 comipiler problems

[Julian Taylor]

On 10/09/2015 06:50 PM, Charles R Harris wrote:

Hi All,

There is a compilation problem
with 1.10.0 on 32 bit
windows using the msvc9 compiler. One possible solution to this is to
drop support for python 2.6. The last, and final, release of of that
series was Python 2.6.9 in Oct 2013. The first release was in 2008.

Thoughts?



doesn't the problem also affect python2.7? I don't recall which msvc is 
required for that but I though it was v9.


If its only the compiler needed for python2.6 thats affected then +1, we 
already dropped binary support for that in numpy 1.9.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

[Julian Taylor]

On 10/08/2015 03:30 PM, David Cournapeau wrote:



On Tue, Oct 6, 2015 at 8:04 PM, Nathaniel Smith > wrote:

On Tue, Oct 6, 2015 at 11:52 AM, David Cournapeau
> wrote:
 >
 >
 > On Tue, Oct 6, 2015 at 7:30 PM, Nathaniel Smith > wrote:
 >>
 >> [splitting this off into a new thread]
 >>
 >> On Tue, Oct 6, 2015 at 3:00 AM, David Cournapeau
>
 >> wrote:
 >> [...]
 >> > I also agree the current situation is not sustainable -- as we
discussed
 >> > privately before, cythonizing numpy.core is made quite more
complicated
 >> > by
 >> > this. I have myself quite a few issues w/ cythonizing the
other parts of
 >> > umath. I would also like to support the static link better
than we do
 >> > now
 >> > (do we know some static link users we can contact to validate our
 >> > approach
 >> > ?)
 >> >
 >> > Currently, what we have in numpy core is the following:
 >> >
 >> > numpy.core.multiarray -> compilation units in
numpy/core/src/multiarray/
 >> > +
 >> > statically link npymath
 >> > numpy.core.umath -> compilation units in numpy/core/src/umath +
 >> > statically
 >> > link npymath/npysort + some shenanigans to use things in
 >> > numpy.core.multiarray
 >>
 >> There are also shenanigans in the other direction - supposedly umath
 >> is layered "above" multiarray, but in practice there are circular
 >> dependencies (see e.g. np.set_numeric_ops).
 >
 > Indeed, I am not arguing about merging umath and multiarray.

Oh, okay :-).

>> > I would suggest to have a more layered approach, to enable both 
'normal'
>> > build and static build, without polluting the public namespace too 
much.
>> > This is an approach followed by most large libraries (e.g. MKL), and is
>> > fairly flexible.
>> >
>> > Concretely, we could start by putting more common functionalities (aka
>> > the
>> > 'core' library) into its own static library. The API would be 
considered
>> > private to numpy (no stability guaranteed outside numpy), and every
>> > exported
>> > symbol from that library would be decorated appropriately to avoid
>> > potential
>> > clashes (e.g. '_npy_internal_').
>>
>> I don't see why we need this multi-layered complexity, though.
>
>
> For several reasons:
>
>  - when you want to cythonize either extension, it is much easier to
> separate it as cython for CPython API, C for the rest.

I don't think this will help much, because I think we'll want to have
multiple cython files, and that we'll probably move individual
functions between being implemented in C and Cython (including utility
functions). So that means we need to solve the problem of mixing C and
Cython files inside a single library.


Separating the pure C code into static lib is the simple way of
achieving the same goal. Essentially, you write:

# implemented in npyinternal.a
_npy_internal_foo()

# implemented in merged_multiarray_umath.pyx
cdef PyArray_Foo(...):
 # use _npy_internal_foo()

then our merged_multiarray_umath.so is built by linking the .pyx and the
npyinternal.a together. IOW, the static link is internal.

Going through npyinternal.a instead of just linking .o from pure C and
Cython together gives us the following:

  1. the .a can just use normal linking strategies instead of the
awkward capsule thing. Those are easy to get wrong when using cython as
you may end up with multiple internal copies of the wrapped object
inside capsule, causing hard to track bugs (this is what we wasted most
of the time on w/ Stefan and Kurt during ds4ds)
  2. the only public symbols in .a are the ones needed by the cython
wrapping, and since those are decorated with npy_internal, clashes are
unlikely to happen
  3. since most of the code is already in .a internally, supporting the
static linking should be simpler since the only difference is how you
statically link the cython-generated code. Because of 1, you are also
less likely to cause nasty surprises when putting everything together.




I don't see why static libraries for internals are discussed at all?
There is not much difference between an .a (archive) file and an .o 
(object) file. What you call a static library is just a collection of 
object files with an index slapped on top for faster lookup.
Whether a symbol is exported or not is defined in the object file, not 
the archive file, so in this regard static library of collection of .o 
files makes no difference.
So our current system also produces a library, the only thing thats 
"missing" is bundling it into an archive via ar cru *.o


I also don't see how pycapsule plays a role in this. You don't need 

Re: [Numpy-discussion] Numpy 1.10.0 release

[Julian Taylor]

On 10/06/2015 01:45 PM, Neal Becker wrote:

Are extra_compile_args actually used in all compile steps?


extra_compile_args is not used by numpy, its to support some third party 
use case I never understood.
As the typical site.cfg used by numpy only contains binaries that are 
never compiled by numpy itself it should have no effect on anything.



CFLAGS='-march=native -O3' python setup.py build

Does seem to use my CFLAGS, as it always did on previous numpy versions.



still seems to work for me, though the preferred variable is OPT= as 
CFLAGS will contain a bunch of other stuff related to building python 
extensions themselves (e.g. -fno-strict-aliasing)


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [SciPy-Dev] Numpy 1.10.0 release

[Julian Taylor]

On 10/06/2015 02:08 PM, Daπid wrote:

I don't get any failures on Fedora 22. I have installed it with pip,
setting my CFLAGS to "-march=core-avx-i -O2 -pipe -mtune=native" and
linking against openblas.

With the new Numpy, Scipy full suite shows two errors, I am sorry I
didn't think of running that in the RC phase:

> ==
> FAIL: test_weighting (test_stats.TestHistogram)

this is a known issue see scipy/scipy/#5148
It can most likely be ignored as the scipy test is too sensitive to 
floating point rounding.



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem while writing array with np.savetxt

[Julian Taylor]

numpy text io is fundamentally broken in python3, there are sometimes
workarounds, but currently its probably best to stick to python2 or not
use it.
The workarounds usually just make actually fixing the functions harder.

On 09/24/2015 09:24 AM, Eric Firing wrote:
> On 2015/09/23 9:17 PM, Andrew Nelson wrote:
>> Dear list,
>> whilst trying to write an array to disk I am coming across the
>> following.  What am I doing wrong?  Surely f is a file handle?
>> (python 3.4.3, numpy 1.9.2)
>>
>> import numpy as np
>> a = np.arange(10.)
>> with open('test.dat', 'w') as f:
>>  np.savetxt(f, a)
>>
> 
> It will work if you open with 'wb'.  Yes, this seems like a bug; or at
> least, the anomaly should be noted in the docstring.
> 
> Eric
> 
>> ---
>>
>>
>> TypeError  Traceback (most recent call
>> last)
>>   in() 2 a = np.arange(10.)3
>> with open('test.dat', 'w') as f:>
>> 4np.savetxt(f,
>> a)/Users/anz/Documents/Andy/programming/dev3/lib/python3.4/site-packages/numpy/lib/npyio.py
>>
>> in savetxt(fname, X, fmt, delimiter, newline, header, footer, comments)
>> 1085 else:1086 for row in X:-> 1087fh.write(asbytes(format % tuple(row)
>> + newline))1088 if len(footer) > 0:1089 footer = footer.replace('\n',
>> '\n' + comments)TypeError: must be str, not bytes
>>
>>
>>
>>
>> -- 
>> _
>> Dr. Andrew Nelson
>>
>>
>> _
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5?

[Julian Taylor]

as due to the many incompatiblities in 1.10 many will likely not be able 
to update anytime soon, so I think putting out another 1.9.3 bugfix 
release would be a good idea.
I can probably do the release management for it, though I haven't been 
keeping up with bugfixes recently so, please comment on important issues 
you would want fixed.
The np.where upcast regression and np.nanmedian issues come to my mind 
as should be fixed.

On 09/14/2015 11:21 AM, Carl Kleffner wrote:
> I would like to add patches for the mingwpy windows build as well. There
> is no Python-3.5 build so far.
>
> Carlkl
>
> 2015-09-14 10:46 GMT+02:00 Robert Kern  >:
>
> On Mon, Sep 14, 2015 at 9:32 AM, Matthew Brett
> > wrote:
> >
> > Hi,
> >
> > On Mon, Sep 14, 2015 at 1:22 AM, David Cournapeau  > wrote:
> > >
> > >
> > > On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett 
> >
> > > wrote:
> > >>
> > >> Hi,
> > >>
> > >> I'm just building numpy 1.9.2 for Python 3.5 (just released).
> > >>
> > >> In order to get the tests to pass on Python 3.5, I need to cherry 
> pick
> > >> commit 7d6aa8c onto the 1.9.2 tag position.
> > >>
> > >> Does anyone object to me uploading a wheel built from this patched
> > >> version to pypi as 1.9.2 for Python 3.5 on OSX?   It would help to 
> get
> > >> the ball rolling for Python 3.5 binary wheels.
> > >
> > >
> > > Why not releasing this as 1.9.3 ? It does not need to be a full 
> release
> > > (with binaries and all), but having multiple sources for a given tag 
> is
> > > confusing.
> >
> > Generally OK with me, but it's quite a bit of extra work for very
> > little gain.  We'd have to tag, release a source tarball and OSX
> > wheels, at least.
>
> I think it's highly desirable that we also have a *source* release
> that builds on Python 3.5, irrespective of whether or not we have
> binary wheels for a couple of platforms up for Python 3.5. So I
> would encourage a quick 1.9.3 release that incorporates this patch.
>
> --
> Robert Kern
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org 
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] SHA256 mismatch on SourceForge downloads

[Julian Taylor]

The file is also not signed so the checksums are not trustworthy anyway. 
Please sign the releases as we did in the past.

On 08/26/2015 10:28 AM, Antoine Pitrou wrote:

 Hello,

 The SourceForge download page for 1.10.0b1 mentions:

   89e467cec774527dd254c1e039801726db1367433053801f0d8bc68deac74009
   numpy-1.10.0b1.tar.gz

 But after downloading the file I get:

 $ sha256sum numpy-1.10.0b1.tar.gz
 855695405092686264dc8ce7b3f5c939a6cf1a5639833e841a5bb6fb799cd6a8
 numpy-1.10.0b1.tar.gz


 Also, since SouceForge doesn't provide any HTTPS downloads (it
 actually redirects HTTPS to HTTP (*)), this all looks a bit pointless.

 (*) seems like SourceForge is becoming a poster child of worst
 practices...

 Regards

 Antoine.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.?

[Julian Taylor]

On 31.07.2015 08:24, Jason Newton wrote:
 Been using numpy in it's various forms since like 2005.  burned on int,
 int_ just today with boost.python / ndarray conversions and a number of
 times before that.  intc being C's int!? Didn't even know it existed
 till today.  This isn't the first time, esp with float.  Bool is
 actually expected for me and I'd prefer it stay 1 byte for storage
 efficiency - I'll use a long if I want it machine word wide.

A long is only machine word wide on posix, in windows its not. This
nonsense is unfortunately also in numpy. It also affects dtype=int.
The correct word size type is actually np.intp.

btw. if something needs deprecating it is np.float128, this is the most
confusing type name in all of numpy as its precision is actually a 80
bit in most cases (x86), 64 bit sometimes (arm) and very rarely actually
128 bit (sparc).
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: Deprecate np.int, np.float, etc.?

[Julian Taylor]

On 07/23/2015 04:29 AM, Nathaniel Smith wrote:
 Hi all,

 So one of the things exposed in the numpy namespace are objects called
np.int
np.float
np.bool
 etc.

 These are commonly used -- in fact, just yesterday on another project
 I saw a senior person reviewing a pull request instruct a more junior
 person that they should use np.float instead of float or np.float64.
 But AFAICT everyone who is actually using them is doing this based on
 a very easy-to-fall-for misconception, i.e., that these objects have
 something to do with numpy.

I don't see the issue. They are just aliases so how is np.float worse 
than just float?
Too me this does not seem worth the bother of deprecation.
An argument could be made for deprecating creating dtypes from python 
builtin types as they are ambiguous (C float != python float) and 
platform dependent. E.g. dtype=int is just an endless source of bugs.
But this is also so invasive that the deprecation would never be 
completed and just be a bother to everyone.

So -1 from me.



 P.S.: using metamodule.py also gives us the option of making
 np.testing lazily imported, which last time this came up was
 benchmarked to improve numpy's import speed by ~35% [1] -- not too bad
 given that most production code will never touch np.testing. But this
 is just a teaser postscript; I'm not proposing that we actually do
 this at this time :-).

 [1] http://mail.scipy.org/pipermail/numpy-discussion/2012-July/063147.html


I doubt these numbers from 2012 are still correct. When this was last 
profiled last year the import there were two main offenders, add_docs 
and np.polynomial. Both have been fixed in 1.9. I don't recall 
np.testing even showing up.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about unaligned access

[Julian Taylor]

On 06.07.2015 18:21, Francesc Alted wrote:
 2015-07-06 18:04 GMT+02:00 Jaime Fernández del Río jaime.f...@gmail.com
 mailto:jaime.f...@gmail.com:
 
 On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted fal...@gmail.com
 mailto:fal...@gmail.com wrote:
 
 Hi,
 
 I have stumbled into this:
 
 In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
 dtype=[('f0', np.int64), ('f1', np.int32)])
 
 In [63]: %timeit sa['f0'].sum()
 100 loops, best of 3: 4.52 ms per loop
 
 In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
 dtype=[('f0', np.int64), ('f1', np.int64)])
 
 In [65]: %timeit sa['f0'].sum()
 1000 loops, best of 3: 896 µs per loop
 
 The first structured array is made of 12-byte records, while the
 second is made by 16-byte records, but the latter performs 5x
 faster.  Also, using an structured array that is made of 8-byte
 records is the fastest (expected):
 
 In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)),
 dtype=[('f0', np.int64)])
 
 In [67]: %timeit sa['f0'].sum()
 1000 loops, best of 3: 567 µs per loop
 
 Now, my laptop has a Ivy Bridge processor (i5-3380M) that should
 perform quite well on unaligned data:
 
 
 http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/
 
 So, if 4 years-old Intel architectures do not have a penalty for
 unaligned access, why I am seeing that in NumPy?  That strikes
 like a quite strange thing to me.
 
 
 I believe that the way numpy is setup, it never does unaligned
 access, regardless of the platform, in case it gets run on one that
 would go up in flames if you tried to. So my guess would be that you
 are seeing chunked copies into a buffer, as opposed to bulk copying
 or no copying at all, and that would explain your timing
 differences. But Julian or Sebastian can probably give you a more
 informed answer.
 
 
 Yes, my guess is that you are right.  I suppose that it is possible to
 improve the numpy codebase to accelerate this particular access pattern
 on Intel platforms, but provided that structured arrays are not that
 used (pandas is probably leading this use case by far, and as far as I
 know, they are not using structured arrays internally in DataFrames),
 then maybe it is not worth to worry about this too much.
 
 Thanks anyway,
 Francesc
  
 
 
 Jaime
  
 
 
 Thanks,
 Francesc
 
 -- 
 Francesc Alted
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 -- 
 (\__/)
 ( O.o)
 (  ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus
 planes de dominación mundial.
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 -- 
 Francesc Alted
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 




signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about unaligned access

[Julian Taylor]

On 06.07.2015 18:21, Francesc Alted wrote:
 2015-07-06 18:04 GMT+02:00 Jaime Fernández del Río jaime.f...@gmail.com
 mailto:jaime.f...@gmail.com:
 
 On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted fal...@gmail.com
 mailto:fal...@gmail.com wrote:
 
 Hi,
 
 I have stumbled into this:
 
 In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
 dtype=[('f0', np.int64), ('f1', np.int32)])
 
 In [63]: %timeit sa['f0'].sum()
 100 loops, best of 3: 4.52 ms per loop
 
 In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
 dtype=[('f0', np.int64), ('f1', np.int64)])
 
 In [65]: %timeit sa['f0'].sum()
 1000 loops, best of 3: 896 µs per loop
 
 The first structured array is made of 12-byte records, while the
 second is made by 16-byte records, but the latter performs 5x
 faster.  Also, using an structured array that is made of 8-byte
 records is the fastest (expected):
 
 In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)),
 dtype=[('f0', np.int64)])
 
 In [67]: %timeit sa['f0'].sum()
 1000 loops, best of 3: 567 µs per loop
 
 Now, my laptop has a Ivy Bridge processor (i5-3380M) that should
 perform quite well on unaligned data:
 
 
 http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/
 
 So, if 4 years-old Intel architectures do not have a penalty for
 unaligned access, why I am seeing that in NumPy?  That strikes
 like a quite strange thing to me.
 
 
 I believe that the way numpy is setup, it never does unaligned
 access, regardless of the platform, in case it gets run on one that
 would go up in flames if you tried to. So my guess would be that you
 are seeing chunked copies into a buffer, as opposed to bulk copying
 or no copying at all, and that would explain your timing
 differences. But Julian or Sebastian can probably give you a more
 informed answer.
 
 
 Yes, my guess is that you are right.  I suppose that it is possible to
 improve the numpy codebase to accelerate this particular access pattern
 on Intel platforms, but provided that structured arrays are not that
 used (pandas is probably leading this use case by far, and as far as I
 know, they are not using structured arrays internally in DataFrames),
 then maybe it is not worth to worry about this too much.
 
 Thanks anyway,
 Francesc
  
 
 
 Jaime
  
 
 
 Thanks,
 Francesc
 
 -- 
 Francesc Alted
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 -- 
 (\__/)
 ( O.o)
 (  ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus
 planes de dominación mundial.
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 -- 
 Francesc Alted
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 




signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about unaligned access

[Julian Taylor]

On 06.07.2015 18:21, Francesc Alted wrote:
 2015-07-06 18:04 GMT+02:00 Jaime Fernández del Río jaime.f...@gmail.com
 mailto:jaime.f...@gmail.com:
 
 On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted fal...@gmail.com
 mailto:fal...@gmail.com wrote:
 
 Hi,
 
 I have stumbled into this:
 
 In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
 dtype=[('f0', np.int64), ('f1', np.int32)])
 
 In [63]: %timeit sa['f0'].sum()
 100 loops, best of 3: 4.52 ms per loop
 
 In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
 dtype=[('f0', np.int64), ('f1', np.int64)])
 
 In [65]: %timeit sa['f0'].sum()
 1000 loops, best of 3: 896 µs per loop
 
 The first structured array is made of 12-byte records, while the
 second is made by 16-byte records, but the latter performs 5x
 faster.  Also, using an structured array that is made of 8-byte
 records is the fastest (expected):
 
 In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)),
 dtype=[('f0', np.int64)])
 
 In [67]: %timeit sa['f0'].sum()
 1000 loops, best of 3: 567 µs per loop
 
 Now, my laptop has a Ivy Bridge processor (i5-3380M) that should
 perform quite well on unaligned data:
 
 
 http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/
 
 So, if 4 years-old Intel architectures do not have a penalty for
 unaligned access, why I am seeing that in NumPy?  That strikes
 like a quite strange thing to me.
 
 
 I believe that the way numpy is setup, it never does unaligned
 access, regardless of the platform, in case it gets run on one that
 would go up in flames if you tried to. So my guess would be that you
 are seeing chunked copies into a buffer, as opposed to bulk copying
 or no copying at all, and that would explain your timing
 differences. But Julian or Sebastian can probably give you a more
 informed answer.
 
 
 Yes, my guess is that you are right.  I suppose that it is possible to
 improve the numpy codebase to accelerate this particular access pattern
 on Intel platforms, but provided that structured arrays are not that
 used (pandas is probably leading this use case by far, and as far as I
 know, they are not using structured arrays internally in DataFrames),
 then maybe it is not worth to worry about this too much.
 
 Thanks anyway,
 Francesc
  
 
 
 Jaime
  
 
 
 Thanks,
 Francesc
 
 -- 
 Francesc Alted
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 -- 
 (\__/)
 ( O.o)
 (  ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus
 planes de dominación mundial.
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 -- 
 Francesc Alted
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 




signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about unaligned access

[Julian Taylor]

sorry for the 3 empty mails, my client bugged out...

as a workaround you can align structured dtypes to avoid this issue:

sa = np.fromiter(((i,i) for i in range(1000*1000)),
dtype=np.dtype([('f0', np.int64), ('f1', np.int32)], align=True))


On 06.07.2015 18:21, Francesc Alted wrote:
 2015-07-06 18:04 GMT+02:00 Jaime Fernández del Río jaime.f...@gmail.com
 mailto:jaime.f...@gmail.com:
 
 On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted fal...@gmail.com
 mailto:fal...@gmail.com wrote:
 
 Hi,
 
 I have stumbled into this:
 
 In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
 dtype=[('f0', np.int64), ('f1', np.int32)])
 
 In [63]: %timeit sa['f0'].sum()
 100 loops, best of 3: 4.52 ms per loop
 
 In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
 dtype=[('f0', np.int64), ('f1', np.int64)])
 
 In [65]: %timeit sa['f0'].sum()
 1000 loops, best of 3: 896 µs per loop
 
 The first structured array is made of 12-byte records, while the
 second is made by 16-byte records, but the latter performs 5x
 faster.  Also, using an structured array that is made of 8-byte
 records is the fastest (expected):
 
 In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)),
 dtype=[('f0', np.int64)])
 
 In [67]: %timeit sa['f0'].sum()
 1000 loops, best of 3: 567 µs per loop
 
 Now, my laptop has a Ivy Bridge processor (i5-3380M) that should
 perform quite well on unaligned data:
 
 
 http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/
 
 So, if 4 years-old Intel architectures do not have a penalty for
 unaligned access, why I am seeing that in NumPy?  That strikes
 like a quite strange thing to me.
 
 
 I believe that the way numpy is setup, it never does unaligned
 access, regardless of the platform, in case it gets run on one that
 would go up in flames if you tried to. So my guess would be that you
 are seeing chunked copies into a buffer, as opposed to bulk copying
 or no copying at all, and that would explain your timing
 differences. But Julian or Sebastian can probably give you a more
 informed answer.
 
 
 Yes, my guess is that you are right.  I suppose that it is possible to
 improve the numpy codebase to accelerate this particular access pattern
 on Intel platforms, but provided that structured arrays are not that
 used (pandas is probably leading this use case by far, and as far as I
 know, they are not using structured arrays internally in DataFrames),
 then maybe it is not worth to worry about this too much.
 
 Thanks anyway,
 Francesc
  
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Open CV 3.0 + NPY_RELAXED_STRIDES

[Julian Taylor]

On Thu, Jun 11, 2015 at 11:39 AM, Sebastian Berg
sebast...@sipsolutions.net wrote:
 On Mi, 2015-06-10 at 21:03 -0600, Charles R Harris wrote:


 snip


   * Relaxed stride checking will be the default in 1.10.0
 Is this still the plan?


 Yes, but it won't be quite the same as the master branch.  Currently
 an unusual value for the stride (?) is used in order to smoke out
 misuse, but that value will be more rational in the release.


 +1, it should not be as bad/common in practice once rolled out. That
 said, I do not mind delaying things beyond 1.10, it might be better for
 compatibility if someone gets a new numpy on top of oldish other
 packages.
 So I am good with planning to go ahead for the moment. But if anyone
 complains, I would back down for 1.10 probably.

 - Sebastian


With beside scipy.ndimage also opencv being broken I think we will
have to delay it beyond 1.10, though we should have at least an alpha,
maybe even a beta with it enabled to induce some panic that hopefully
will spure some fixes.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Verify your sourceforge windows installer downloads

[Julian Taylor]

On 28.05.2015 19:46, Pauli Virtanen wrote:
 28.05.2015, 20:35, Sturla Molden kirjoitti:
 Pauli Virtanen p...@iki.fi wrote:

 Is it possible to host them on github? I think there's an option to add
 release notes and (apparently) to upload binaries if you go to the
 Releases section --- there's one for each tag.

 And then Sourceforge will put up tainted installers for the benefit of
 NumPy users. :)
 
 Well, let them. They may already be tainted, who knows. It's phishing
 and malware distribution at that point, and there are some ways to deal
 with that (safe browsing, AV etc).
 
 

there is no guarantee that github will not do this stuff in future too,
also PyPI or self hosting do not necessarily help as those resources can
be compromised.
The main thing that should be learned this and the many similar
incidents in the past is that binaries from the internet need to be
verified of they have been modified from their original state otherwise
they cannot be trusted.

With my mail I wanted to bring to attention that both numpy (since
1.7.2) and scipy (since 0.14.1) allow users to do so via the signed
README.txt containing checksums.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Verify your sourceforge windows installer downloads

[Julian Taylor]

hi,
It has been reported that sourceforge has taken over the gimp
unofficial windows downloader page and temporarily bundled the
installer with unauthorized adware:
https://plus.google.com/+gimp/posts/cxhB1PScFpe

As NumPy is also distributing windows installers via sourceforge I
recommend that when you download the files you verify the downloads
via the checksums in the README.txt before using them. The README.txt
is clearsigned with my gpg key so it should be safe from tampering.
Unfortunately as I don't use windows I cannot give any advice on how
to do the verifcation on these platforms. Maybe someone familar with
available tools can chime in.

I have checked the numpy downloads and they still match what I
uploaded, but as sourceforge does redirect based on OS and geolocation
this may not mean much.

Cheers,
Julian Taylor
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Strategy for OpenBLAS

[Julian Taylor]

On 05/26/2015 04:56 PM, Matthew Brett wrote:
 Hi,
 
 This morning I was wondering whether we ought to plan to devote some
 resources to collaborating with the OpenBLAS team.
 
 
 
 It is relatively easy to add tests using Python / numpy.  We like
 tests.  Why don't we propose a collaboration with OpenBLAS where we
 build and test numpy with every / most / some commits of OpenBLAS, and
 try to make it easy for the OpenBLAS team to add tests.Maybe we
 can use and add to the list of machines on which OpenBLAS is tested
 [1]?  We Berkeley Pythonistas can certainly add the machines at our
 buildbot farm [2].  Maybe the Julia / R developers would be interested
 to help too?
 

Technically we only need a single machine with the newest instruction
set available. All other cases could then be tested via a virtual
machine that only exposes specific instruction sets (e.g. qemu which
could technically also emulate stuff the host does not have).

Concerning test generation there is a huge parameter space that needs
testing due with openblas, at least some of it would need to be
automated/fuzzed. We also need specific preconditioning of memory to
test failure cases openblas had in the past, E.g. filling memory around
the matrices with nans and also somehow filling openblas own temporary
buffers with some signaling values (might require special built openblas
if _MALLOC_PERTURB does not work).

Maybe it would be feasible to write a hypothesis [0] strategy for some
of the blas stuff to automate the parameter exploration.

And then we'd need to run everything under valgrind as due to the
assembler implementation of openblas we can't use the faster address
sanitizers gcc and clang now provide.

[0] https://hypothesis.readthedocs.org/en/latest/
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Using gentxt to import a csv with a string class label and hundreds of integer features

[Julian Taylor]

On Thu, May 7, 2015 at 2:26 AM, Dammy damilarefagb...@gmail.com wrote:
 Hi,
 I am trying to use numpy.gentxt to import a csv for classification using
 scikit-learn. The first column in the csv is a string type class label while
 200+ extra columns are integer features.
 Please I wish to find out how I can use the gentext function to specify a
 dtype of string for the first column while specifying int type for all other
 columns.

 I have tried using dtype=None as shown below, but when I print
 dataset.shape,  I get (number_or_rows,) i.e no columns are read in:
  dataset = np.genfromtxt(file,delimiter=',', skip_header=True)

 I also tried setting the dtypes as shown in the examples below, but I get
 the same error as dtype=None:

these dtypes will create structured arrays:
http://docs.scipy.org/doc/numpy/user/basics.rec.html

so it is expected that the shape is the number of rows, the colums are
part of the dtype and can be accessed like a dictionary:

In [21]: d = np.ones(3, dtype='S2, int8')

In [22]: d
Out[22]:
array([('1', 1), ('1', 1), ('1', 1)],
  dtype=[('f0', 'S2'), ('f1', 'i1')])

In [23]: d.shape
Out[23]: (3,)

In [24]: d.dtype.names
Out[24]: ('f0', 'f1')

In [25]: d[0]
Out[25]: ('1', 1)

In [26]: d['f0']
Out[26]:
array(['1', '1', '1'],
  dtype='|S2')

In [27]: d['f1']
Out[27]: array([1, 1, 1], dtype=int8)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] performance of numpy.array()

[Julian Taylor]

numpy 1.9 makes array(list) performance similar in performance to vstack
in 1.8 its very slow.

On 29.04.2015 17:40, simona bellavista wrote:
 on cluster A 1.9.0 and on cluster B 1.8.2
 
 2015-04-29 17:18 GMT+02:00 Nick Papior Andersen nickpap...@gmail.com
 mailto:nickpap...@gmail.com:
 
 Compile it yourself to know the limitations/benefits of the
 dependency libraries.
 
 Otherwise, have you checked which versions of numpy they are, i.e.
 are they the same version?
 
 2015-04-29 17:05 GMT+02:00 simona bellavista afy...@gmail.com
 mailto:afy...@gmail.com:
 
 I work on two distinct scientific clusters. I have run the same
 python code on the two clusters and I have noticed that one is
 faster by an order of magnitude than the other (1min vs 10min,
 this is important because I run this function many times). 
 
 I have investigated with a profiler and I have found that the
 cause of this is that (same code and same data) is the function
 numpy.array that is being called 10^5 times. On cluster A it
 takes 2 s in total, whereas on cluster B it takes ~6 min.  For
 what regards the other functions, they are generally faster on
 cluster A. I understand that the clusters are quite different,
 both as hardware and installed libraries. It strikes me that on
 this particular function the performance is so different. I
 would have though that this is due to a difference in the
 available memory, but actually by looking with `top` the memory
 seems to be used only at 0.1% on cluster B. In theory numpy is
 compiled with atlas on cluster B, and on cluster A it is not
 clear, because numpy.__config__.show() returns NOT AVAILABLE for
 anything.
 
 Does anybody has any insight on that, and if I can improve the
 performance on cluster B?
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 -- 
 Kind regards Nick
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] performance of numpy.array()

[Julian Taylor]

On 29.04.2015 17:50, Robert Kern wrote:
 On Wed, Apr 29, 2015 at 4:05 PM, simona bellavista afy...@gmail.com
 mailto:afy...@gmail.com wrote:

 I work on two distinct scientific clusters. I have run the same python
 code on the two clusters and I have noticed that one is faster by an
 order of magnitude than the other (1min vs 10min, this is important
 because I run this function many times).

 I have investigated with a profiler and I have found that the cause of
 this is that (same code and same data) is the function numpy.array that
 is being called 10^5 times. On cluster A it takes 2 s in total, whereas
 on cluster B it takes ~6 min.  For what regards the other functions,
 they are generally faster on cluster A. I understand that the clusters
 are quite different, both as hardware and installed libraries. It
 strikes me that on this particular function the performance is so
 different. I would have though that this is due to a difference in the
 available memory, but actually by looking with `top` the memory seems to
 be used only at 0.1% on cluster B. In theory numpy is compiled with
 atlas on cluster B, and on cluster A it is not clear, because
 numpy.__config__.show() returns NOT AVAILABLE for anything.

 Does anybody has any insight on that, and if I can improve the
 performance on cluster B?
 
 Check to see if you have the Transparent Hugepages (THP) Linux kernel
 feature enabled on each cluster. You may want to try turning it off. I
 have recently run into a problem with a large-memory multicore machine
 with THP for programs that had many large numpy.array() memory
 allocations. Usually, THP helps memory-hungry applications (you can
 Google for the reasons), but it does require defragmenting the memory
 space to get contiguous hugepages. The system can get into a state where
 the memory space is so fragmented such that trying to get each new
 hugepage requires a lot of extra work to create the contiguous memory
 regions. In my case, a perfectly well-performing program would suddenly
 slow down immensely during it's memory-allocation-intensive actions.
 When I turned THP off, it started working normally again.
 
 If you have root, try using `perf top` to see what C functions in user
 space and kernel space are taking up the most time in your process. If
 you see anything like `do_page_fault()`, this, or a similar issue, is
 your problem.
 

this issue it has nothing to do with thp, its a change in array in numpy
1.9. Its now as fast as vstack, while before it was really really slow.

But the memory compaction is indeed awful, especially the backport
redhat did for their enterprise linux.

Typically it is enough to only disable the automatic defragmentation on
allocation only, not the full thps, e.g. via
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
(on redhat backports its a different path)

You still have the hugepaged running defrags at times of low load and in
limited fashion, you can also manually trigger a defrag by writting to:
/prog/sys/vm/compact_memory
Though the hugepaged which runs only occasionally should already do a
good job.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Asking proposal review/feedback for GSOC 15

[Julian Taylor]

On 03/26/2015 12:58 PM, Oğuzhan Ünlü wrote:
 Hi,
 
 Sorry for a bit late reply. I will express my thoughts for Ralf's
 suggestions, respectively.
  
 
 Regarding your schedule:
 - I would remove the parts related to benchmarks. There's no nice
 benchmark
 infrastructure in numpy itself at the moment (that's a separate GSoC
 idea),
 so the two times 1 week that you have are likely not enough to get
 something off the ground there.
 
  
 - I think we can do a sample/demo benchmark only based on a library'
 speed performance over some basic set of data sets. Couldn't we? Instead
 of speed, it could be any other performance parameter, we can decide
 together.

Creating benchmark and performance tracking tools should not be part of
this project, but benchmarking is still important.

You may have to research learn how to best benchmark this low level
code, understand what influences their performance and we need a good
set of benchmarks so we know in the end what we have gained by this project.
I think the time allocation for this is good.

  
 
 - The implement a flexible interface part will need some discussion,
 probably it makes sense to first draft a document (call it a NEP - Numpy
 Enhancement Proposal) that lays out the options and makes a proposal.
 
  
 To be realistic, I don't think I have enough time to complete an
 enhancement proposal. Maybe we can talk about it in the first half of
 April?  

I think he means writing the nep proposal should be part of the project,
you don't need to have a fleshed out one ready now. Though if you
already have some ideas on how the interface might look like this should
go into your proposal.


 
 - I wouldn't put investigate accuracy differences at the end. What
 if you
 find out there that you've been working on something for the whole
 summer
 that's not accurate enough?
 
 
 However, we can't examine possible accuracy differences without having
 seen their real performance (in my case it is 'implementing an interface
 to libraries'). Isn't investigating possible libraries for numpy the
 fountain head of this project? Integrating chosen library can be
 possible by a small set of wrapping functions.

The accuracy of the libraries can be investigated prior to their
integration into numpy, and it should be done early to rule out or
de-prioritize bad options.
Documenting the trade-offs between performance and accuracy is one of
the most important tasks.

This also involves researching what kind of inputs to functions may be
numerically problematic which depending on your prior numerics knowledge
may take some time and should be accounted for.



  
 
 - The researching possible options I would do in the community bonding
 period - when the coding period starts you should have a fairly
 well-defined plan.
 
 
 I agree with you at this point. After moving this to community bounding
 period, I can put a milestone like 'integrating chosen library to numpy'
 for 2 weeks. And we decide it would be better to remove benchmark part,
 then I would use that part for interface, probably. 
  
 
 - 3 weeks for implementing the interface looks optimistic.
 
 
 It was an estimated time, I asked Julian's opinion about it and waiting
 his answer. You could be right, I am not familiar with codebase and
 exact set of functions to be improved. Since I prepared my schedule to
 serve as basis, I think it is understandable if something takes a bit
 longer or shorter as compared to what is written on schedule. 
 

I am pretty bad as estimating times, but I think the implementation of
interfaces can be done in three weeks if you are confident enough in
your C coding abilities and are some experienced in maneuvering foreign
code bases.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] element-wise array segmental function operation?

[Julian Taylor]

On 23.03.2015 07:46, oyster wrote:
 Hi, all
 I want to know wether there is a terse way to apply a function to
 every array element, where the function behaves according to the
 element value.
 for example
 [code]
 def fun(v):
 if 0=v60:
 return f1(v)#where f1 is a function
 elif 60=v70:
 return f2(v)
 elif 70=v80:
 return f3(v)
 ...and so on...
 [/code]
 
 for 'a=numpy.array([20,50,75])', I hope to get numpy.array([f1(20),
 f1(50), f3(75)])
 

piecewise should be what you are looking for:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.piecewise.html


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Mathematical functions in Numpy

[Julian Taylor]

currently the math functions are wrapped via the generic PyUfunc_*
functions in numpy/core/src/umath/loops.c.src which just apply some
arbitrary function to a scalar from arbitrarily strided inputs.
When adding variants one likely needs to add some special purpose loops
to deal with the various special requirements of the vector math api's.
This involves adding some special cases to the ufunc generation in
numpy/core/code_generators/generate_umath.py and then implementing the
new kernel functions.
See e.g. this oldish PR, which changes the sqrt function from a
PyUfunc_d_d function to a special loop to take advantage of the
vectorized machine instructions:
https://github.com/numpy/numpy/pull/3341

some things have changed a bit since then but it does show many of the
files you probably need to look for this project.

On 17.03.2015 19:51, Robert Kern wrote:
 On Tue, Mar 17, 2015 at 6:29 PM, Matthieu Brucher
 matthieu.bruc...@gmail.com mailto:matthieu.bruc...@gmail.com wrote:

 Hi,

 These functions are defined in the C standard library!
 
 I think he's asking how to define numpy ufuncs.
 
 2015-03-17 18:00 GMT+00:00 Shubhankar Mohapatra
 mshubhan...@yahoo.co.in mailto:mshubhan...@yahoo.co.in:
  Hello all,
  I am a undergraduate and i am trying to do a project this time on
 numppy in
  gsoc. This project is about integrating vector math library classes
 of sleef
  and yeppp into numpy to make the mathematical functions faster. I have
  already studied the new library classes but i am unable to find the
 sin ,
  cos function definitions in the numpy souce code.Can someone please
 help me
  find the functions in the source code so that i can implement the new
  library class into numpy.
  Thanking you,
  Shubhankar Mohapatra
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 



 --
 Information System Engineer, Ph.D.
 Blog: http://matt.eifelle.com
 LinkedIn: http://www.linkedin.com/in/matthieubrucher
 Music band: http://liliejay.com/
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 --
 Robert Kern
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Introductory mail and GSoc Project Vector math library integration

[Julian Taylor]

On 03/12/2015 10:15 AM, Gregor Thalhammer wrote:
 
 Another note, numpy makes it easy to provide new ufuncs, see 
 http://docs.scipy.org/doc/numpy-dev/user/c-info.ufunc-tutorial.html
 from a C function that operates on 1D arrays, but this function needs to
 support arbitrary spacing (stride) between the items. Unfortunately, to
 achieve good performance, vector math libraries often expect that the
 items are laid out contiguously in memory. MKL/VML is a notable
 exception. So for non contiguous in- or output arrays you might need to
 copy the data to a buffer, which likely kills large amounts of the
 performance gain.

The elementary functions are very slow even compared to memory access,
they take in the orders of hundreds to tens of thousand cycles to
complete (depending on range and required accuracy).
Even in the case of strided access that gives the hardware prefetchers
plenty of time to load the data before the previous computation is done.

This also removes the requirement from the library to provide a strided
api, we can copy the strided data into a contiguous buffer and pass it
to the library without losing much performance. It may not be optimal
(e.g. a library can fine tune the prefetching better for the case where
the hardware is not ideal) but most likely sufficient.

Figuring out how to best do it to get the best performance and still
being flexible in what implementation is used is part of the challenge
the student will face for this project.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy array casting ruled not safe

[Julian Taylor]

On 08.03.2015 11:49, Sebastian Berg wrote:
 On Sa, 2015-03-07 at 18:21 -0800, Jaime Fernández del Río wrote:
 snip
 
 
 
 
 I note that on SO Jaime made the suggestion that take use
 unsafe casting and throw an error on out of bounds indexes.
 That sounds reasonable, although for sufficiently large
 integer types an index could wrap around to a good value.
 Maybe make it work only for npy_uintp.
 
 
 Chuck  


 It is mostly about consistency, and having take match what indexing
 already does, which is to unsafely cast all integers:


 In [11]: np.arange(10)[np.uint64(2**64-1)]
 Out[11]: 9


 I think no one has ever complained about that obviously wrong
 behavior, but people do get annoyed if they cannot use their perfectly
 valid uint64 array because we want to protect them from themselves.
 Sebastian has probably given this more thought than anyone else, it
 would be interesting to hear his thoughts on this.

 
 Not really, there was no change in behaviour for arrays here. Apparently
 though (which I did not realize), there was a change for numpy
 scalars/0-d arrays. Of course I think ideally same_type casting would
 raise an error or at least warn on out of bounds integers, but we do not
 have a mechanism for that.
 
 We could fix this, I think Jaime you had thought about that at some
 point? But it would require loop specializations for every integer type.
 
 So, I am not sure what to prefer, but for the user indexing with
 unsigned integers has to keep working without explicit cast. Of course
 the fact that it is dangerous, is bothering me a bit, even if a
 dangerous wrap-around seems unlikely in practice.
 


I was working on supporting arbitrary integer types as index without
casting.
This would have a few advantages, you can save memory without
sacrificing indexing performance by using smaller integers and you can
skip the negative index wraparound step for unsigned types.
But it does add quite a bit of code bloat that is essentially a
micro-optimization.
To make it really useful one also needs to adapt other functions like
where, arange, meshgrid, indices etc. to have an option to return the
smallest integer type that is sufficient for the index array.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

[Julian Taylor]

On 07.03.2015 00:20, Pauli Virtanen wrote:
 06.03.2015, 22:43, Eric Firing kirjoitti:
 On 2015/03/06 10:23 AM, Pauli Virtanen wrote:
 06.03.2015, 20:00, Benjamin Root kirjoitti:
 A slightly different way to look at this is one of sharing data. If I am
 working on a system with 3.4 and I want to share data with others who may
 be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
 less attractive.

 pickle is used in npy files only if there are object arrays in them.
 Of course, savez could just decline saving object arrays.

 Or issue a prominent warning.
 
 https://github.com/numpy/numpy/pull/5641
 

I think the ship for a warning has long sailed. At this point its
probably more an annoyance for python3 users and will not prevent many
more python2 users from saving files that can't be loaded into python3.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PR, extended site.cfg capabilities

[Julian Taylor]

On 02/24/2015 02:16 PM, Nick Papior Andersen wrote:
 Dear all,
 
 I have initiated a PR-5597 https://github.com/numpy/numpy/pull/5597,
 which enables the reading of new flags from the site.cfg file.
 @rgommers requested that I posted some information on this site,
 possibly somebody could test it on their setup.

I do not fully understand the purpose of these changes. Can you give
some more detailed use cases?


 
 So the PR basically enables reading these extra options in each section:
 runtime_library_dirs : Add runtime library directories to the shared
 libraries (overrides the dreaded LD_LIBRARY_PATH)

LD_LIBRARY_PATH should not be used during compilation, this is a runtime
flag that numpy.distutils has no control over.
Can you explain in more detail what you intend to do with this flag?

 extra_compile_args: Adds extra compile flags to the compilation

extra flags to which compilation?
site.cfg lists libraries that already are compiled. The only influence
compiler flags could have is for header only libraries that profit from
some flags. But numpy has no support for such libraries currently. E.g.
cblas.h (which is just a header with signatures) is bundled with numpy.
I guess third parties may make use of this, an example would be good.

 extra_link_args: Adds extra flags when linking to libraries

This flag may be useful.
It could be used to pass options required during linking, like
-Wl,--no-as-needed that is sometimes needed to link with gsl.
Possibly also useful for link time optimizations.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PR, extended site.cfg capabilities

[Julian Taylor]

On 02/24/2015 02:31 PM, Julian Taylor wrote:
 On 02/24/2015 02:16 PM, Nick Papior Andersen wrote:
 Dear all,

 I have initiated a PR-5597 https://github.com/numpy/numpy/pull/5597,
 which enables the reading of new flags from the site.cfg file.
 @rgommers requested that I posted some information on this site,
 possibly somebody could test it on their setup.
 
 I do not fully understand the purpose of these changes. Can you give
 some more detailed use cases?

I think I understand better now, so this is intended as a site.cfg
equivalent (and possibly more portable) variant of the environment
variables that control these options?
e.g. runtime_lib_dirs would be equivalent to LD_RUN_PATH env. variable
and build_ext --rpath
and the compile_extra_opts equivalent to the OPT env variable?

 
 

 So the PR basically enables reading these extra options in each section:
 runtime_library_dirs : Add runtime library directories to the shared
 libraries (overrides the dreaded LD_LIBRARY_PATH)
 
 LD_LIBRARY_PATH should not be used during compilation, this is a runtime
 flag that numpy.distutils has no control over.
 Can you explain in more detail what you intend to do with this flag?
 
 extra_compile_args: Adds extra compile flags to the compilation
 
 extra flags to which compilation?
 site.cfg lists libraries that already are compiled. The only influence
 compiler flags could have is for header only libraries that profit from
 some flags. But numpy has no support for such libraries currently. E.g.
 cblas.h (which is just a header with signatures) is bundled with numpy.
 I guess third parties may make use of this, an example would be good.
 
 extra_link_args: Adds extra flags when linking to libraries
 
 This flag may be useful.
 It could be used to pass options required during linking, like
 -Wl,--no-as-needed that is sometimes needed to link with gsl.
 Possibly also useful for link time optimizations.
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GSoC'15 - mentors ideas

[Julian Taylor]

On 02/24/2015 05:41 PM, Charles R Harris wrote:
 
 
 On Mon, Feb 23, 2015 at 11:49 PM, Ralf Gommers ralf.gomm...@gmail.com
 mailto:ralf.gomm...@gmail.com wrote:
 
 Hi all,
 
 
 On Fri, Feb 20, 2015 at 10:05 AM, Ralf Gommers
 ralf.gomm...@gmail.com mailto:ralf.gomm...@gmail.com wrote:
 
 Hi all,
 
 It's time to start preparing for this year's Google Summer of
 Code. There is actually one urgent thing to be done (before
 19.00 UTC today), which is to get our ideas page in decent
 shape. It doesn't have to be final, but there has to be enough
 on there for the organizers to judge it. This page is here:
 https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be
 reworking it and linking it from the PSF page today, but if you
 already have new ideas please add them there. See
 https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate
 for this year's template for adding a new idea.
 
 
 The ideas page is now in pretty good shape. More ideas are very
 welcome though, especially easy or easy/intermediate ideas. Numpy
 right now has zero easy ones and Scipy only one and a half.
 
 
 We could add a benchmark project for numpy that would build off the work
 Pauli is doing in Scipy. That would be easy to intermediate I think, as
 the programming bits might be easy, but coming up with the benchmarks
 would be more difficult.
 

we already have decent set of benchmarks in yaroslavs setup:
http://yarikoptic.github.io/numpy-vbench/

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.nonzero behavior with multidimensional arrays

[Julian Taylor]

On 23.02.2015 08:52, Jaime Fernández del Río wrote:
 This was raised in SO today:
 
 http://stackoverflow.com/questions/28663142/why-is-np-wheres-result-read-only-for-multi-dimensional-arrays/28664009
 
 np.nonzero (and np.where for boolean arrays) behave differently for 1-D
 and higher dimensional arrays:
 
 In the first case, a tuple with a single behaved base ndarray is returned:
 
 In the second, a tuple with as many arrays as dimensions in the passed
 array is returned, but the arrays are not base ndarrays, but of the same
 subtype as was passed to the function. These arrays are also set as
 non-writeable:
 


The non-writeable looks like a bug too me, it should probably just use
PyArray_FLAGS(self) instead of 0. We had a similar one with the new
indexing, its easy to forget this.

Concerning subtypes, I don't think there is a good reason to preserve
them here and it should just return an ndarray.
where with one argument returns a new object that indexes the input
object so it is not really related anymore to what it indexes and there
is no information that numpy could reasonably propagate.

(where with three arguments make sense with subtypes and fixing that is
on my todo list)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vectorizing computation

[Julian Taylor]

On 02/13/2015 11:51 AM, Francesc Alted wrote:
 Hi,
 
 I would like to vectorize the next computation:
 
 nx, ny, nz = 720, 180, 3
 outheight = np.arange(nz) * 3
 oro = np.arange(nx * ny).reshape((nx, ny))
 
 def compute1(outheight, oro):
 result = np.zeros((nx, ny, nz))
 for ix in range(nx):
 for iz in range(nz):
 result[ix, :, iz] = outheight[iz] + oro[ix, :]
 return result
 
 I think this should be possible by using an advanced use of broadcasting
 in numpy.  Anyone willing to post a solution?


result = outheight + oro.reshape(nx, ny, 1)


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vectorizing computation

[Julian Taylor]

On 02/13/2015 01:03 PM, Francesc Alted wrote:
 2015-02-13 12:51 GMT+01:00 Julian Taylor jtaylor.deb...@googlemail.com
 mailto:jtaylor.deb...@googlemail.com:
 
 On 02/13/2015 11:51 AM, Francesc Alted wrote:
  Hi,
 
  I would like to vectorize the next computation:
 
  nx, ny, nz = 720, 180, 3
  outheight = np.arange(nz) * 3
  oro = np.arange(nx * ny).reshape((nx, ny))
 
  def compute1(outheight, oro):
  result = np.zeros((nx, ny, nz))
  for ix in range(nx):
  for iz in range(nz):
  result[ix, :, iz] = outheight[iz] + oro[ix, :]
  return result
 
  I think this should be possible by using an advanced use of
 broadcasting
  in numpy.  Anyone willing to post a solution?
 
 
 result = outheight + oro.reshape(nx, ny, 1)
 
 
 And 4x faster for my case.  Oh my, I am afraid that my mind will never
 scratch all the amazing possibilities that broadcasting is offering :)
 
 Thank you very much for such an elegant solution!
  


if speed is a concern this is faster as it has a better data layout for
numpy during the computation, but the result may be worse layed out for
further processing

result = outheight.reshape(nz, 1, 1) + oro
return np.rollaxis(result, 0, 3)


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Aligned / configurable memory allocation

[Julian Taylor]

On 10.02.2015 22:33, Nathaniel Smith wrote:
 On 10 Feb 2015 13:10, Antoine Pitrou solip...@pitrou.net
 mailto:solip...@pitrou.net wrote:

 On Tue, 10 Feb 2015 11:26:22 -0800
 Nathaniel Smith n...@pobox.com mailto:n...@pobox.com wrote:
  On 10 Feb 2015 09:11, Antoine Pitrou solip...@pitrou.net
 mailto:solip...@pitrou.net wrote:
  
  
   Hello,
  
   I apologize for pinging the list, but I was wondering if there was
   interest in either of https://github.com/numpy/numpy/pull/5457 (make
   array data aligned by default) or
   https://github.com/numpy/numpy/pull/5470 (make the array data
 allocator
   configurable)?
 
  I'm not a fan of the configurable allocator. It adds new public APIs
 for us
  to support, and makes switching to using Python's own memory allocation
  APIs more complex. The feature is intrinsically dangerous, because newly
  installed deallocators must be able to handle memory allocated by the
  previous allocator. (AFAICT the included test case can crash the test
  process if you get unlucky and GC runs during it?).

 It's taken care of in the patch.

unfortunately it also breaks the ABI on two fronts, by adding a new
member to the public array struct which needs initializing by non api
using users and by removing the ability to use free on array pointers.
Both not particularly large breaks, but breaks nonetheless.

At least for the first issue we should (like for the proposed dtype and
ufunc changes) apply a more generic break of hiding the new internal
members in a new private structure that embeds the public structure
unchanged.
The second issue can probably be ignored, though we could retain it for
posix/c11 as those standards wisely decided to make aligned pointers
freeable with free.
That on the other hand costs us efficient calloc and realloc (standard
comities are weird sometimes ...)


 
 Ah, I see -- I missed that you added an allocator field to
 PyArrayObject. That does reduce my objections to the patch. But I'm
 still not sure what problems this is solving exactly.
 
 Also, if we do decide to add a deallocation callback to PyArrayObject
 then I think we should take advantage of the opportunity to also make
 life easier for c API users who need a custom callback on a case-by-case
 basis and currently have to jump through hoops using -base.
 



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: NumPy 1.9.2 release candidate

[Julian Taylor]

Hi,

We have finished the first release candidate of NumPy 1.9.2.
The 1.9.2 release will as usual be a bugfix only release to the 1.9.x
series.
The tarballs and win32 binaries are available on sourceforge:
https://sourceforge.net/projects/numpy/files/NumPy/1.9.2rc1/

If no regressions show up the final release is planned next week.
The upgrade is recommended for all users of the 1.9.x series.

Following issues have been fixed:
* gh-5316: fix too large dtype alignment of strings and complex types
* gh-5424: fix ma.median when used on ndarrays
* gh-5481: Fix astype for structured array fields of different byte order
* gh-5155: Fix loadtxt with comments=None and a string None data
* gh-4476: Masked array view fails if structured dtype has datetime
component
* gh-5388: Make RandomState.set_state and RandomState.get_state threadsafe
* gh-5390: make seed, randint and shuffle threadsafe
* gh-5374: Fixed incorrect assert_array_almost_equal_nulp documentation
* gh-5393: Add support for ATLAS  3.9.33.
* gh-5313: PyArray_AsCArray caused segfault for 3d arrays
* gh-5492: handle out of memory in rfftf
* gh-4181: fix a few bugs in the random.pareto docstring
* gh-5359: minor changes to linspace docstring
* gh-4723: fix a compile issues on AIX

Source tarballs, windows installers and release notes can be found at
https://sourceforge.net/projects/numpy/files/NumPy/1.9.2rc1/

Cheers,
The NumPy Developer team



signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Sorting refactor

[Julian Taylor]

On 16.01.2015 12:33, Lars Buitinck wrote:
 2015-01-16 11:55 GMT+01:00  numpy-discussion-requ...@scipy.org:
 Message: 2
 Date: Thu, 15 Jan 2015 21:24:00 -0800
 From: Jaime Fern?ndez del R?o jaime.f...@gmail.com
 Subject: [Numpy-discussion] Sorting refactor
 To: Discussion of Numerical Python numpy-discussion@scipy.org
 Message-ID:
 capowhwkf6rnwcrgmcwsmq_lo3hshjgbvlsrn19z-mdpe25e...@mail.gmail.com
 Content-Type: text/plain; charset=utf-8

 This changes will make it easier for me to add a Timsort generic type
 function to numpy's arsenal of sorting routines. And I think they have
 value by cleaning the source code on their own.
 
 Yes, they do. I've been looking at the sorting functions as well and
 I've found the following:
 
 * The code is generally hard to read because it prefers pointer over
 indices. I'm wondering if it would get slower using indices. The
 closer these algorithms are to the textbook, the easier to insert
 fancy optimizations.
 
 * The heap sort exploits undefined behavior by using a pointer that
 points before the start of the array. However, rewriting it to always
 point within the array made it slower. I haven't tried rewriting it
 using indices.
 
 * Quicksort has a quadratic time worst case. I think it should be
 turned into an introsort [1] for O(n log n) worst case; we have the
 heapsort needed to do that.

This probably rarely happens in numeric data, and we do have guaranteed
nlog runtime algorithms available.
But it also is not costly to do, e.g. the selection code is a
introselect instead of a normal quickselect.
I'd say not high priority, but if someone wants to do it I don't see why
not.

 
 * Quicksort is robust to repeated elements, but doesn't exploit them.
 It can be made to run in linear time if the input array has only O(1)
 distinct elements [2]. This may come at the expense of some
 performance on arrays with no repeated elements.
 
 * Using optimal sorting networks instead of insertion sort as the base
 case can speed up quicksort on float arrays by 5-10%, but only if NaNs
 are moved out of the way first so that comparisons become cheaper [3].
 This has consequences for the selection algorithms that I haven't
 fully worked out yet.

I was also thinking about this, an advantage of a sorting network is
that it can be vectorized to be significantly faster than an insertion
sort. Handling NaN's should also be directly possible.
The issue is that its probably too much complicated code for only a very
small gain.

 
 * Using Cilk Plus to parallelize merge sort can make it significantly
 faster than quicksort at the expense of only a few lines of code, but
 I haven't checked whether Cilk Plus plays nicely with multiprocessing
 and other parallelism options (remember the trouble with OpenMP-ified
 OpenBLAS?).

you should also be able to do this with openmp tasks, though it will be
a little less efficient as cilk+ has a better scheduler for this type of
work.
But I assume you will get the same trouble as openmp but that needs
testing, also cilk+ in gcc is not really production ready yet, I got
lots of crashes when I last tried it (it will be in 5.0 though).


 
 This isn't really an answer to your questions, more like a brain dump
 from someone who's stared at the same code for a while and did some
 experiments. I'm not saying we should implement all of this, but keep
 in mind that there are some interesting options besides implementing
 timsort.
 
 [1] https://en.wikipedia.org/wiki/Introsort
 [2] http://www.sorting-algorithms.com/static/QuicksortIsOptimal.pdf
 [3] https://github.com/larsmans/numpy/tree/sorting-nets
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Build doesn't pass tests

[Julian Taylor]

On 01/08/2015 07:31 PM, Nikolay Mayorov wrote:
 Hi all! 
 
 I'm trying to build numpy on Windows 64 bit, Python 3.4.2 64 bit.  
 
 I do environment setup by the following command:
 
 CMD /K SET MSSdk=1  SET DISTUTILS_USE_SDK=1  C:\Program Files
 (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat x86_amd64
 
 Then I cd to the newly cloned numpy folder and do: python setup.py
 build_ext --inplace 
 
 It looks like the build process finishes correctly. 
 
 But then python -c import numpy; numpy.test() crashes the interpreter
 (some tests pass before the crash). I found out that it is caused by
 numpy.fromfile function call.
 
 What might be the reason of that? Do I use wrong msvc compiler?
 
 

I think compiling python3 extensions requires VS 2010, python 2
extensions VS2008.
A crash in a fromfile test is what I would expect from using the wrong
compiler.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fast sizes for FFT

[Julian Taylor]

I still have the plan to add this function as public api to numpy's fft
helper functions, though I didn't get to it yet.
Its a relative simple task if someone wants to contribute.

On 24.12.2014 04:33, Robert McGibbon wrote:
 Alex Griffing pointed out on github that this feature was recently added
 to scipy in https://github.com/scipy/scipy/pull/3144. Sweet!
 
 -Robert
 
 On Tue, Dec 23, 2014 at 6:47 PM, Charles R Harris
 charlesr.har...@gmail.com mailto:charlesr.har...@gmail.com wrote:
 
 
 
 On Tue, Dec 23, 2014 at 7:32 PM, Robert McGibbon rmcgi...@gmail.com
 mailto:rmcgi...@gmail.com wrote:
 
 Hey,
 
 The performance of fftpack depends very strongly on the array
 size -- sizes that are powers of two are good, but also powers
 of three, five and seven, or numbers whose only prime factors
 are from (2,3,5,7). For problems that can use padding, rounding
 up the size (using np.fft.fft(x, n=size_with_padding)) to one of
 these multiples makes a big difference.
 
 Some other packages expose a function for calculating the next
 fast size, e.g: http://ltfat.sourceforge.net/notes/ltfatnote017.pdf.
 
 Is there anything like this in numpy/scipy? If not, would this
 be a reasonable feature to add?
 
 
 It would be nice to have, but an integrated system would combine it
 with padding and windowing. Might be worth putting together a
 package, somewhat like seaborn for plotting, that provides  a nicer
 interface to the fft module. Tracking downsampling/upsampling  and
 units would also be useful. I don't know if anyone has done
 something like that already...
 
 Chuck
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fast sizes for FFT

[Julian Taylor]

On 24.12.2014 13:07, Sturla Molden wrote:
 On 24/12/14 04:33, Robert McGibbon wrote:
 Alex Griffing pointed out on github that this feature was recently added
 to scipy in https://github.com/scipy/scipy/pull/3144. Sweet!
 
 
 I use different padsize search than the one in SciPy. It would be 
 interesting to see which is faster.
 

hm this is a brute force search, probably fast enough but slower than
scipy's code (if it also were cython)

I also ported it to C a while back, so that could be used for numpy if
speed is an issue. Code attached.

static inline unsigned long get_msb(unsigned long a)
{
/* dumb integer msb, (63 - __builtin_clzl(a)) would be faster */
unsigned long msb = 0;
while (a = 1)  {
msb++;
}
return msb;
}

/* ---*/
/**
 * @brief  get next 5-smooth number next to a certain value
 * @param a  lower bound of result
 *
 * 5-smooth number is a number with only prime factors 2, 3 or 5.
 * This can be used to get minimal padding sizes that still allow efficient fft
 * transforms.
 */
/* ---*/
size_t visir_get_next_regular(size_t a)
{
/* fftw can also deal efficiently with factors of 7 and 13 but it needs
 * testing that the speed of these algorithms will not cancel out the gain
 * of smaller input sizes */
if (a = 6)
return a;
/* is already power of 2 */
if ((a  (a - 1)) == 0)
return a;
/* overflow */
if (5  SIZE_MAX / a)
return a;

size_t match = SIZE_MAX;
size_t p5 = 1;
while(p5  a) {
size_t p35 = p5;
while (p35  a) {
/* ceil division */
size_t quotient = a % p35 != 0 ? a / p35 + 1 : a / p35;
/* next power of two of quotient */
size_t p2 = 2  get_msb(quotient - 1);

size_t n = p2 * p35;
if (n == a)
return n;
else if (n  match)
match = n;

p35 *= 3;
if (p35 == a)
return p35;
}
if (p35  match)
match = p35;
p5 *= 5;
if (p5 == a)
return p5;
}
if (p5  match)
match = p5;
return match;
}

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] simple reduction question

[Julian Taylor]

On 24.12.2014 16:25, Neal Becker wrote:
 What would be the most efficient way to compute:
 
 c[j] = \sum_i (a[i] * b[i,j])
 
 where a[i] is a 1-d vector, b[i,j] is a 2-d array?
 
 This seems to be one way:
 
 import numpy as np
 a = np.arange (3)
 b = np.arange (12).reshape (3,4)
 c = np.dot (a, b).sum()
 
 but np.dot returns a vector, which then needs further reduction.  Don't know 
 if 
 there's a better way.
 

the formula maps nice to einsum:

np.einsum(i,ij-, a, b)

should also be reasonably efficient, but that probably depends on your
BLAS library and the sizes of the arrays.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Context manager for seterr

[Julian Taylor]

On 15.12.2014 01:12, Stefan van der Walt wrote:
 Hi all,
 
 Since the topic of context managers recently came up, what do you think
 of adding a context manager for seterr?
 
 with np.seterr(divide='ignore'):
 frac = num / denom
 


already exists as np.errstate:

with np.errstate(divide='ignore'):
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Context manager for seterr

[Julian Taylor]

On 15.12.2014 01:40, Stefan van der Walt wrote:
 On 2014-12-15 02:23:18, Julian Taylor jtaylor.deb...@googlemail.com wrote:
 already exists as np.errstate:

 with np.errstate(divide='ignore'):
 
 With 'ignore' a warning is still raised--is this by choice?
 
 import numpy as np
 x = np.array([0, 1, 2.])
 with np.errstate(divide='ignore'):
 ... x/x
 ... 
 __main__:2: RuntimeWarning: invalid value encountered in true_divide
 array([ nan,   1.,   1.])
 
 
 (I see it is documented that way as well, so I suspect so.)
 

0./0. raises an invalid floating point exception, unlike e.g 1./0. which
raises a zero division exception.
NumPy just bubbles up what the processor does, which means it does not
behave like Python which always raises ZeroDivision also for 0./0.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] help using np.correlate to produce correlograms.

[Julian Taylor]

I think it is a good time to discuss/implement further correlate
improvements.
I kind of favor the mode=(tuple of integers) api for your proposed change.
Concerning the C-API we probably need to add a new wrapper function but
thats ok, the C-API does not need to be as nice as the python API as it
has far less and typically more experienced users.

I also think its time we can remove the old_behavior flag which has been
there since 1.4.
Are there objections to that?

Also on a side note, in 1.10 np.convolve/correlate has been
significantly speed up if one of the sequences is less than 12 elements
long.

On 12/11/2014 09:54 AM, Pierre Haessig wrote:
 As a side note, I've still in mind the proposal I made back in 2013 to 
 make np.correlate faster
 
 http://numpy-discussion.10968.n7.nabble.com/another-discussion-on-numpy-correlate-and-convolution-td32925.html
 
 The basic idea is to enable the user to select the exact range of lags 
 he wants. Unfortunately I didn't take the time to go further than the 
 specification above...
 
 best,
 Pierre
 
 
 Le 10/12/2014 21:40, Jose Guzman a écrit :
 Dear Pierre,

 thank you very much for your time to correct my notebook and to point me
 in the direction of my wrong lag estimation. It has been very useful!

 Best

 Jose

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] help using np.correlate to produce correlograms.

[Julian Taylor]

On 12/11/2014 03:24 PM, Pierre Haessig wrote:
 Le 11/12/2014 11:19, Julian Taylor a écrit :
 Also on a side note, in 1.10 np.convolve/correlate has been
 significantly speed up if one of the sequences is less than 12 elements
 Interesting! What is the origin of this speed up, and why a magic number 12?
 

previously numpy called dot for the convolution part, this is fine for
large convolutions as dot goes out to BLAS which is superfast.
For small convolutions unfortunately it is terrible as generic dot in
BLAS libraries have enormous overheads they only amortize on large data.
So one part was computing the dot in a simple numpy internal loop if the
data is small.

The second part is the number of registers typical machines have, e.g.
amd64 has 16 floating point registers. If you can put all elements of a
convolution kernel into these registers you save reloading them from
stack on each iteration.
11 is the largest number I could reliably use without the compiler
spilling them to the stack.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] help using np.correlate to produce correlograms.

[Julian Taylor]

On 11.12.2014 19:01, Jose Guzman wrote:
 On 11/12/14 09:54, Pierre Haessig wrote:
 The basic idea is to enable the user to select the exact range of lags
 he wants. Unfortunately I didn't take the time to go further than the
 specification above...
 
 I would be particularly interested in computing cross-correlations in a 
 range of +-4000 sampling points lags. Unfortunately, my 
 cross-correlations require vectors of ~8e6 of points, and np.correlate 
 performs very slowly if I compute the whole range.
 
 I also heard that a faster alternative to compute the cross-correlation 
 is to perform the product of the Fourier transform of the 2 vectors and 
 then performing the inverse Fourier of the result.
 


Large convolutions/correlations are generally faster in fourier space as
they have O(NlogN) instead of O(N^2) complexity, for 1e6 points this
should be very significant.
You can use scipy.signal.fftconvolve to do that conveniently (with
performance optimal zero padding). Convolution of a flipped input (and
conjugated?) is the same as a correlation.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should ndarray be a context manager?

[Julian Taylor]

I don't think that makes much sense, context managers are useful for
managing the lifetime of objects owning resources not already managed by
the garbage collector.
E.g. file descriptors, a gc has no clue that a piece of memory contains
a descriptor and thus never has a reason to release it in time when
there is plenty of memory available.

Memory on the other hand is the resource a gc manages, so it should
release objects when memory pressure is high.

Also numpy only supports CPython so we don't even need to care about
that. A context manager will also not help you with reference cycles.

On 09.12.2014 16:01, Sturla Molden wrote:
 
 I wonder if ndarray should be a context manager so we can write 
 something like this:
 
 
 with np.zeros(n) as x:
[...]
 
 
 The difference should be that __exit__ should free the memory in x (if 
 owned by x) and make x a zero size array.
 
 Unlike the current ndarray, which does not have an __exit__ method, this 
 would give precise control over when the memory is freed. The timing of 
 the memory release would not be dependent on the Python implementation, 
 and a reference cycle or reference leak would not accidentally produce a 
 memory leak. It would allow us to deterministically decide when the 
 memory should be freed, which e.g. is useful when we work with large arrays.
 
 
 A problem with this is that the memory in the ndarray would be volatile 
 with respect to other Python threads and view arrays. However, there are 
 dozens of other ways to produce segfaults or buffer overflows with NumPy 
 (cf. stride_tricks or wrapping external buffers).
 
 
 Below is a Cython class that does something similar, but we would need 
 to e.g. write something like
 
  with Heapmem(n * np.double().itemsize) as hm:
  x = hm.doublearray
  [...]
 
 instead of just
 
  with np.zeros(n) as x:
  [...]
 
 
 Sturla
 
 
 # (C) 2014 Sturla Molden
 
 from cpython cimport PyMem_Malloc, PyMem_Free
 from libc.string cimport memset
 cimport numpy as cnp
 cnp.init_array()
 
 
 cdef class Heapmem:
 
  cdef:
  void *_pointer
  cnp.intp_t _size
 
  def __cinit__(Heapmem self, Py_ssize_t n):
  self._pointer = NULL
  self._size = cnp.intp_t n
 
  def __init__(Heapmem self, Py_ssize_t n):
  self.allocate()
 
  def allocate(Heapmem self):
  if self._pointer != NULL:
  raise RuntimeError(Memory already allocated)
  else:
  self._pointer = PyMem_Malloc(self._size)
  if (self._pointer == NULL):
  raise MemoryError()
  memset(self._pointer, 0, self._size)
 
  def __dealloc__(Heapmem self):
  if self._pointer != NULL:
  PyMem_Free(self._pointer)
  self._pointer = NULL
 
  property pointer:
  def __get__(Heapmem self):
  return cnp.intp_t self._pointer
 
  property doublearray:
  def __get__(Heapmem self):
  cdef cnp.intp_t n = self._size//sizeof(double)
  if self._pointer != NULL:
  return cnp.PyArray_SimpleNewFromData(1, n,
   cnp.NPY_DOUBLE, self._pointer)
  else:
  raise RuntimeError(Memory not allocated)
 
  property chararray:
  def __get__(Heapmem self):
  if self._pointer != NULL:
  return cnp.PyArray_SimpleNewFromData(1, self._size,
   cnp.NPY_CHAR, self._pointer)
  else:
  raise RuntimeError(Memory not allocated)
 
  def __enter__(self):
  if self._pointer != NULL:
  raise RuntimeError(Memory not allocated)
 
  def __exit__(Heapmem self, type, value, traceback):
  if self._pointer != NULL:
  PyMem_Free(self._pointer)
  self._pointer = NULL
 
 
 
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GDB macros for numpy

[Julian Taylor]

On Sun, Nov 30, 2014 at 10:41 PM, David Cournapeau courn...@gmail.com
wrote:



 On Sun, Nov 30, 2014 at 5:45 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:



 On Sun, Nov 30, 2014 at 4:54 AM, David Cournapeau courn...@gmail.com
 wrote:

 Hi there,

 I remember having seen some numpy-aware gdb macros at some point, but
 cannot find any reference. Does anyone know of any ?


 What would numpy aware gdb macros be? Never heard of them.


 Python itself has some gdb macros (
 https://wiki.python.org/moin/DebuggingWithGdb).

 You can do things like:

 # p is a PyObject * pointer to the list [1, 2]
  pyo p
 [1, 2]

 The idea would be to have a nice representation for arrays, dtypes, etc...

 David


this wiki entry is very very old or not aware of the official python gdb
macros (which are built into gdb in typical linux distributions, no extra
configuration required).
and the macros are awesome, they are extremely helpful in debugging python
extension codes.
The current commands are:
py-list
py-bt
py-print
py-locals
py-up/py-down

having similar macros for numpy types might be useful, but at least I more
often than not one does want the actual C structure than a pretty
representation which is just *(type*)PyArray_DATA(arr)@10 away.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GDB macros for numpy

[Julian Taylor]

On Mon, Dec 1, 2014 at 1:44 AM, Julian Taylor jtaylor.deb...@googlemail.com
 wrote:


 On Sun, Nov 30, 2014 at 10:41 PM, David Cournapeau courn...@gmail.com
 wrote:



 On Sun, Nov 30, 2014 at 5:45 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:



 On Sun, Nov 30, 2014 at 4:54 AM, David Cournapeau courn...@gmail.com
 wrote:

 Hi there,

 I remember having seen some numpy-aware gdb macros at some point, but
 cannot find any reference. Does anyone know of any ?


 What would numpy aware gdb macros be? Never heard of them.


 Python itself has some gdb macros (
 https://wiki.python.org/moin/DebuggingWithGdb).

 You can do things like:

 # p is a PyObject * pointer to the list [1, 2]
  pyo p
 [1, 2]

 The idea would be to have a nice representation for arrays, dtypes, etc...

 David


 this wiki entry is very very old or not aware of the official python gdb
 macros (which are built into gdb in typical linux distributions, no extra
 configuration required).


ups I scrolled to far to the legacy section, the earlier parts do mention
the new macros :)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Finding values in an array

[Julian Taylor]

On 28.11.2014 04:15, Alexander Belopolsky wrote:
 I probably miss something very basic, but how given two arrays a and b,
 can I find positions in a where elements of b are located?  If a were
 sorted, I could use searchsorted, but I don't want to get valid
 positions for elements that are not in a.  In my case, a has unique
 elements, but in the general case I would accept the first match.  In
 other words, I am looking for an array analog of list.index() method.

np.where(np.in1d(a, b))
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Finding values in an array

[Julian Taylor]

On 28.11.2014 09:37, Robert Kern wrote:
 On Fri, Nov 28, 2014 at 8:22 AM, Julian Taylor
 jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com
 wrote:

 On 28.11.2014 04:15, Alexander Belopolsky wrote:
  I probably miss something very basic, but how given two arrays a and b,
  can I find positions in a where elements of b are located?  If a were
  sorted, I could use searchsorted, but I don't want to get valid
  positions for elements that are not in a.  In my case, a has unique
  elements, but in the general case I would accept the first match.  In
  other words, I am looking for an array analog of list.index() method.

 np.where(np.in1d(a, b))
 
 Only if the matching elements in `b` have the same order as they do in `a`.
 

seems to work also if unordered:
In [32]: a = np.arange(1000)

In [33]: b = np.arange(500,550, 3)

In [34]: np.random.shuffle(a)

In [35]: np.random.shuffle(b)

In [36]: np.where(np.in1d(a, b))
Out[36]:
(array([  0, 106, 133, 149, 238, 398, 418, 498, 533, 541, 545, 589, 634,
798, 846, 891, 965]),)


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Setting up a newcomers label on the issue tracker ?

[Julian Taylor]

On 11/26/2014 09:44 AM, David Cournapeau wrote:
 Hi,
 
 Would anybody mind if I create a label newcomers on GH, and start
 labelling simple issues ?
 
 This is in anticipation to the bloomberg lab event in London this WE. I
 will try to give a hand to people interested in numpy/scipy,
 
 David
 

we have the easy-fix tag which we use for this purpose, though we have
not been applying it very systematically, the bug list could probably do
with a new sweep for these issues.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release

[Julian Taylor]

On 11/26/2014 02:19 PM, Charles R Harris wrote:
 
 
 On Wed, Nov 26, 2014 at 2:06 AM, Julian Taylor
 jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com
 wrote:
 
 On 11/26/2014 12:50 AM, Andrea Gavana wrote:
 
  On 25 November 2014 at 19:33, David Cournapeau courn...@gmail.com 
 mailto:courn...@gmail.com
  mailto:courn...@gmail.com mailto:courn...@gmail.com wrote:
 
 
 
  On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden
  sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com
 mailto:sturla.mol...@gmail.com mailto:sturla.mol...@gmail.com
 wrote:
 
  David Cournapeau courn...@gmail.com mailto:courn...@gmail.com
  mailto:courn...@gmail.com mailto:courn...@gmail.com wrote:
   Shall we consider a
   
 href=https://github.com/scipy/scipy/issues/4168;https://github.com/scipy/scipy/issues/4168/a
   to be a
   blocker (the issue arises on scipy master as well as 0.14.1) ?
  
 
  It is really bad, but does anyone know what is really going on?
 
 
  Yes, it is in the bug report.
 
 
 
  Nice move.
 
  I've now recommended to hold back any upgrade/update/pip-crap/enpkg-fun
  thing on NumPy/SciPy across the whole user base of Python in the
  company. We will probably move to 64bit-in-any-sense soon enough, I
  guess before this issue is solved. Tell me, NumPy, was the array aligned
  enough in 1.8? Is NumPy stricter in its checking because of SPARC? 
 SPARC?!?
 
 yes, before the change numpy accepted a factor 10 performance regression
 in complex indexing to satisfy sparc.
 
 
  Dozens of f2py compiled extensions are going to fail soon here - which
  I'll reluctantly check tomorrow. I don't want to think about other
  people on Win32 facing the same issue elsewhere... :-)
 
 
 There haven't been any real complaints from applications yet, only
 testsuite failure of scipy.
 Eithe the one thing that is broken in scipy isn't much used or windows
 32 users aren't using 1.9 yet.
 The majority of f2py should still be working, numpys own f2py testsuite
 passes on win32. I still don't know what exactly arpack is doing
 different but I also did not have time yet to look at the testcase david
 created.
 
 It should of course be fixed, I have an old branch with aligned
 allocators lying around somewhere which should fix the issue or at least
 give users a way to work around it.
 
 
 The lesson I take from this is that we need a test ;) Also, that it
 would be nice if we got more testing by users before releases. But
 things are as they are, this will get fixed, and we will move on with
 one more lesson learned.
 
 If anyone is to blame, it is the reviewer, namely, myself.
 


concerning actually fixing it I guess we have 3 options:

1. reduce maximum copy alignment from currently 16 to 8 on platforms
that need it.
That will automatically reduce the needed alignment of complex on win32
to 8 bytes. Do other compilers provide something similar to gccs
__BIGGEST_ALIGNMENT__?
2. remove bloating of complex alignment and let sparc crash.
3. add an aligned allocator

I somewhat favor 1, it has the risk that a vectorizing compiler might
wreak havoc but to my knowledge numpy does not actually have real 16
byte copies. There are some occurrences of npy_int128, but those likely
are not used on windows.

fwiw I could reproduce the issue on linux i386 with -malign-double, (I
could have sworn I tested that configuration too...)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.9.1, zeros and alignement

[Julian Taylor]

On 18.11.2014 19:20, David Cournapeau wrote:
 Hi,
 
 I have not followed closely the changes that happen in 1.9.1, but was
 surprised by the following:
 
 x = np.zeros(12, d)
 assert x.flags.aligned # fails
 
 This is running numpy 1.9.1 built on windows with VS 2008. Is it
 expected that zeros may return a non-aligned array ?
 

what is the real alignment of the array? Are you on 32 bit or 64 bit?
What is the alignment of doubles in windows (linux its 4 byte on 32 bit
8 byte on 64 bit (% special compiler flags)?
print x.__array_interface__[data]

there are problems with complex types but doubles should be aligned even
on 32 bit.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.9.1, zeros and alignement

[Julian Taylor]

 1.9 lies about alignment it doesn't actually check for new arrays.

is the array aligned?

On 18.11.2014 19:37, David Cournapeau wrote:
 Additional point: it seems to always return aligned data on 1.8.1 (same
 platform/compiler/everything).
 
 On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau courn...@gmail.com
 mailto:courn...@gmail.com wrote:
 
 It is on windows 32 bits, but I would need to make this work for
 complex (pair of double) as well.
 
 Is this a bug (I  assumed array creation methods would always create
 aligned arrays for their type) ? Seems like quite a bit of code out
 there would assume this (scipy itself does for example).
 
 (the context is  100 test failures on scipy 0.14.x on top of numpy
 1.9., because f2py intent(inout) fails on work arrays created by
 zeros, this is a windows-32 only failure).
 
 David
 
 On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor
 jtaylor.deb...@googlemail.com
 mailto:jtaylor.deb...@googlemail.com wrote:
 
 On 18.11.2014 19:20, David Cournapeau wrote:
  Hi,
 
  I have not followed closely the changes that happen in 1.9.1,
 but was
  surprised by the following:
 
  x = np.zeros(12, d)
  assert x.flags.aligned # fails
 
  This is running numpy 1.9.1 built on windows with VS 2008. Is it
  expected that zeros may return a non-aligned array ?
 
 
 what is the real alignment of the array? Are you on 32 bit or 64
 bit?
 What is the alignment of doubles in windows (linux its 4 byte on
 32 bit
 8 byte on 64 bit (% special compiler flags)?
 print x.__array_interface__[data]
 
 there are problems with complex types but doubles should be
 aligned even
 on 32 bit.
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.9.1, zeros and alignement

[Julian Taylor]

32 bit windows should not provide 16 byte alignment, at least it doesn't
for me. That is typically a property of 64 bit OS.

But that does not explain why normal double is not aligned for you, that
only needs 4 bytes on i386 which even 32 bit OS should provide.

I though I tested scipy on 32 bit linux which has very similar
properties to win32, time to rerun the test.

On 18.11.2014 19:56, David Cournapeau wrote:
 
 
 On Tue, Nov 18, 2014 at 6:40 PM, Julian Taylor
 jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com
 wrote:
 
  1.9 lies about alignment it doesn't actually check for new arrays.
 
 
 When I do the following on 1.8.1 with win 32 bits:
 
 x = np.zeros(12, D)
 print x.aligned.flags  == (x.__array_interface__[data][0] % 16 == 0) #
 always true
 print x.aligned.flags # always true
 
 but on 1.9.1:
 
 x = np.zeros(12, D)
 print x.aligned.flags == (x.__array_interface__[data][0] % 16 == 0) #
 always true
 print x.aligned.flags # not always true
 
 I wonder why numpy 1.8.1 always returned 16 bytes aligned arrays in this
 case (unlikely to be a coincidence, as I try quite a few times with
 different sizes).
  
 
 
 is the array aligned?
 
 On 18.11.2014 19:37, David Cournapeau wrote:
  Additional point: it seems to always return aligned data on 1.8.1 (same
  platform/compiler/everything).
 
  On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau courn...@gmail.com 
 mailto:courn...@gmail.com
  mailto:courn...@gmail.com mailto:courn...@gmail.com wrote:
 
  It is on windows 32 bits, but I would need to make this work for
  complex (pair of double) as well.
 
  Is this a bug (I  assumed array creation methods would always create
  aligned arrays for their type) ? Seems like quite a bit of code out
  there would assume this (scipy itself does for example).
 
  (the context is  100 test failures on scipy 0.14.x on top of numpy
  1.9., because f2py intent(inout) fails on work arrays created by
  zeros, this is a windows-32 only failure).
 
  David
 
  On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor
  jtaylor.deb...@googlemail.com 
 mailto:jtaylor.deb...@googlemail.com
  mailto:jtaylor.deb...@googlemail.com
 mailto:jtaylor.deb...@googlemail.com wrote:
 
  On 18.11.2014 19:20, David Cournapeau wrote:
   Hi,
  
   I have not followed closely the changes that happen in 1.9.1,
  but was
   surprised by the following:
  
   x = np.zeros(12, d)
   assert x.flags.aligned # fails
  
   This is running numpy 1.9.1 built on windows with VS 2008. Is 
 it
   expected that zeros may return a non-aligned array ?
  
 
  what is the real alignment of the array? Are you on 32 bit or 64
  bit?
  What is the alignment of doubles in windows (linux its 4 byte on
  32 bit
  8 byte on 64 bit (% special compiler flags)?
  print x.__array_interface__[data]
 
  there are problems with complex types but doubles should be
  aligned even
  on 32 bit.
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
 mailto:NumPy-Discussion@scipy.org
 mailto:NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: NumPy 1.9.1 bugfix release

[Julian Taylor]

Hello,

We am pleased to announce the release of NumPy 1.9.1, a
bugfix only release for the 1.9.x series.
https://sourceforge.net/projects/numpy/files/NumPy/1.9.1/
The upgrade is recommended for all users of the 1.9.x series.

Following issues have been fixed:
* gh-5184: restore linear edge behaviour of gradient to as it was in  1.9.
  The second order behaviour is available via the `edge_order` keyword
* gh-4007: workaround Accelerate sgemv crash on OSX 10.9
* gh-5100: restore object dtype inference from iterable objects without
`len()`
* gh-5163: avoid gcc-4.1.2 (red hat 5) miscompilation causing a crash
* gh-5138: fix nanmedian on arrays containing inf
* gh-5240: fix not returning out array from ufuncs with subok=False set
* gh-5203: copy inherited masks in MaskedArray.__array_finalize__
* gh-2317: genfromtxt did not handle filling_values=0 correctly
* gh-5067: restore api of npy_PyFile_DupClose in python2
* gh-5063: cannot convert invalid sequence index to tuple
* gh-5082: Segmentation fault with argmin() on unicode arrays
* gh-5095: don't propagate subtypes from np.where
* gh-5104: np.inner segfaults with SciPy's sparse matrices
* gh-5251: Issue with fromarrays not using correct format for unicode arrays
* gh-5136: Import dummy_threading if importing threading fails
* gh-5148: Make numpy import when run with Python flag '-OO'
* gh-5147: Einsum double contraction in particular order causes ValueError
* gh-479: Make f2py work with intent(in out)
* gh-5170: Make python2 .npy files readable in python3
* gh-5027: Use 'll' as the default length specifier for long long
* gh-4896: fix build error with MSVC 2013 caused by C99 complex support
* gh-4465: Make PyArray_PutTo respect writeable flag
* gh-5225: fix crash when using arange on datetime without dtype set
* gh-5231: fix build in c99 mode


The source distributions have been uploaded to PyPI. The Windows
installers, documentation and release notes can be found at:
https://sourceforge.net/projects/numpy/files/NumPy/1.9.1/

Cheers,
Julian Taylor



signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Deprecate pkgload, PackageLoader

[Julian Taylor]

On 29.10.2014 05:30, Charles R Harris wrote:
 Hi All,
 
 It is proposed to deprecate, then remove, pkgload and PackageLoader.
 
 Complaints? Cries of Anguish?
 

I don't mind the deprecation, but I have to ask why? is it causing issues?
it does look like something some people use in their workflows.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Memory efficient alternative for np.loadtxt and np.genfromtxt

[Julian Taylor]

On 28.10.2014 21:24, Nathaniel Smith wrote:
 On 28 Oct 2014 20:10, Chris Barker chris.bar...@noaa.gov
 mailto:chris.bar...@noaa.gov wrote:

 Memory efficiency -- somethign like my growable array is not all that
 hard to implement and pretty darn quick -- you just do the usual trick_
 over allocate a bit of memory, and when it gets full re-allocate a
 larger chunk.
 
 Can't you just do this with regular numpy using .resize()? What does
 your special class add? (Just curious.)
 
 From a quick loo, it seems that the Panda's code is pretty nice --
 maybe the 2X memory footprint should be ignored.
 
 +1
 
 It's fun to sit around and brainstorm clever implementation strategies,
 but Wes already went ahead and implemented all the tricky bits, and
 optimized them too. No point in reinventing the wheel.
 

just to through it in there, astropy recently also added a faster ascii
file reader:
https://groups.google.com/forum/#!topic/astropy-dev/biCgb3cF0v0
not familiar with how it compares to pandas.

how is pandas support for unicode text files?
unicode is the big weak point of numpys current text readers and needs
to addressed.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: NumPy 1.9.1 release candidate

[Julian Taylor]

Hi,

We have finally finished the first release candidate of NumOy 1.9.1,
sorry for the week delay.
The 1.9.1 release will as usual be a bugfix only release to the 1.9.x
series.
The tarballs and win32 binaries are available on sourceforge:
https://sourceforge.net/projects/numpy/files/NumPy/1.9.1rc1/

If no regressions show up the final release is planned next week.
The upgrade is recommended for all users of the 1.9.x series.

Following issues have been fixed:
* gh-5184: restore linear edge behaviour of gradient to as it was in  1.9.
  The second order behaviour is available via the `edge_order` keyword
* gh-4007: workaround Accelerate sgemv crash on OSX 10.9
* gh-5100: restore object dtype inference from iterable objects without
`len()`
* gh-5163: avoid gcc-4.1.2 (red hat 5) miscompilation causing a crash
* gh-5138: fix nanmedian on arrays containing inf
* gh-5203: copy inherited masks in MaskedArray.__array_finalize__
* gh-2317: genfromtxt did not handle filling_values=0 correctly
* gh-5067: restore api of npy_PyFile_DupClose in python2
* gh-5063: cannot convert invalid sequence index to tuple
* gh-5082: Segmentation fault with argmin() on unicode arrays
* gh-5095: don't propagate subtypes from np.where
* gh-5104: np.inner segfaults with SciPy's sparse matrices
* gh-5136: Import dummy_threading if importing threading fails
* gh-5148: Make numpy import when run with Python flag '-OO'
* gh-5147: Einsum double contraction in particular order causes ValueError
* gh-479: Make f2py work with intent(in out)
* gh-5170: Make python2 .npy files readable in python3
* gh-5027: Use 'll' as the default length specifier for long long
* gh-4896: fix build error with MSVC 2013 caused by C99 complex support
* gh-4465: Make PyArray_PutTo respect writeable flag
* gh-5225: fix crash when using arange on datetime without dtype set
* gh-5231: fix build in c99 mode

Source tarballs, windows installers and release notes can be found at
https://sourceforge.net/projects/numpy/files/NumPy/1.9.1rc1/

Cheers,
Julian Taylor





signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] segfault in np.arange

[Julian Taylor]

On 23.10.2014 19:21, Dave Hirschfeld wrote:
 Hi,
 I accidentally passed a pandas DatetimeIndex to `np.arange` which caused 
 it to segfault. It's a pretty dumb thing to do but I don't think it 
 should cause a segfault!


thanks for the report, this patch should fix it:

https://github.com/numpy/numpy/pull/5225
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy log and exceptions

[Julian Taylor]

On 22.10.2014 05:52, Daniel Hyams wrote:
 
 I would have thought that this snippet would raise an exception:
 
 import numpy
 numpy.seterr(all='raise')
 a = numpy.array([1.0,0.0,-1.0])
 b = numpy.log(a)
 
 I get as a result (in b): [0, -Inf, NaN]
 
 It's basically the same issue as:
 
 http://numpy-discussion.10968.n7.nabble.com/numpy-log-does-not-raise-
 exceptions-td5854.html
 
 Except that I have explicitly set the error flags to raise exceptions.  It 
 works fine for sqrt(), but not for log().  I've checked numpy 1.4.0 and 
 1.7.1 and both have the same behavior.
 
 Is there a way to force the log (and log10) function to raise an exception 
 on invalid input?
 

What platform are you using?
whether you get exceptions or not depends on your math library.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Bug in 1.9?

[Julian Taylor]

On 22.10.2014 20:00, Charles R Harris wrote:
 
 
 On Wed, Oct 22, 2014 at 11:32 AM, Neil Girdhar mistersh...@gmail.com
 mailto:mistersh...@gmail.com wrote:
 
 Hello,
 
 Is this desired behaviour or a regression or a bug?
 
 
 http://stackoverflow.com/questions/26497656/how-do-i-align-a-numpy-record-array-recarray
 
 Thanks,
 
 
 I'd guess that the definition of aligned may have become stricter,
 that's the only thing I think has changed. Maybe Julian can comment on that.
 

structured dtypes have not really a well defined alignment, e.g. the
stride of this is 12, so when element 0 is aligned element 1 is always
unaligned.

Before 1.9 structured dtype always had the aligned flag set, even if
they were unaligned.
Now we require a minimum alignment of 16 for strings and structured
types so copying which sometimes works on the whole compound type
instead of each item always works.
This was the easiest way to get the testsuite running on sparc after
fixing a couple of code paths not updating alignment information which
forced some functions to always take super slow unaligned paths (e.g.
ufunc.at)
But the logic could certainly be improved.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Extract Indices of Numpy Array Based on Given Bit Information

[Julian Taylor]

On 18.10.2014 07:58, Artur Bercik wrote:
 Dear Python and Numpy Users:
 
 My data are in the form of '32-bit unsigned integer' as follows:
 
 myData = np.array([1073741824, 1073741877, 1073742657, 1073742709,
 1073742723, 1073755137, 1073755189,1073755969],dtype=np.int32)
 
 I want to get the index of my data where the following occurs:
 
 Bit No. 0–1
 Bit Combination: 00
 
 How can I do it? I heard this type of problem first time, please help me.
 
 Artur
 

not sure I understand the problem, maybe this?

np.where((myData  0x3) == 0)

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Extract Indices of Numpy Array Based on Given Bit Information

[Julian Taylor]

On 18.10.2014 14:14, Artur Bercik wrote:
 
 
 On Sat, Oct 18, 2014 at 9:00 PM, Artur Bercik vbubbl...@gmail.com
 mailto:vbubbl...@gmail.com wrote:
 
 
 
 On Sat, Oct 18, 2014 at 8:28 PM, Julian Taylor
 jtaylor.deb...@googlemail.com
 mailto:jtaylor.deb...@googlemail.com wrote:
 
 On 18.10.2014 07:58, Artur Bercik wrote:
  Dear Python and Numpy Users:
 
  My data are in the form of '32-bit unsigned integer' as follows:
 
  myData = np.array([1073741824, 1073741877, 1073742657, 1073742709,
  1073742723, 1073755137, 1073755189,1073755969],dtype=np.int32)
 
  I want to get the index of my data where the following occurs:
 
  Bit No. 0–1
  Bit Combination: 00
 
  How can I do it? I heard this type of problem first time, please 
 help me.
 
  Artur
 
 
 not sure I understand the problem, maybe this?
 
 np.where((myData  0x3) == 0)
 
 
 yes, it works greatly for the following case:
 
 myData = np.array([1073741824, 1073741877, 1073742657, 1073742709,
 1073742723, 1073755137, 1073755189,1073755969],dtype=np.uint32)
 Bit No. 0–1
 Bit Combination: 00
 
 Can you make such automation for the following case as well?
 
 Bit No. 2–5
 Bit Combination: 1101
 

sure, you can do any of these with the right masks:
np.where((myData  0x3c) == 0x34)

you can use bin(number) to check if your numbers are correct.


 
 Also wondering why np.where((myData  0x3) == 0) instead of
 just np.where((myData  3) == 0) 
 

its the same, 0x means the number is in hexadecimal representation, for
3 they happen to be equal (as 3  10)
It is often easier to work in the hexadecimal representation when
dealing with binary data as its base is a power of two. So two digits in
hexadecimal represent one byte.
In the case above: 0x3c
c is 12 - 1100
3 is 3  - 11
together you get 00, mask for bits 2-5
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion