Re: [Numpy-discussion] Is this a known error in Ipython with numpy?

[Elliot Hallmark]

Hey Alex. Thanks. I was aware of that. However, I was simply doing `run
myscript.py` on the first input line of the Ipython shell, so I did not
expect this behaviour.

The ipython list would be a better place to ask I guess, since the
behaviour on numpy's part is to be expected. Just wondering if any numpy
folks had seen this before.

Elliot
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is this a known error in Ipython with numpy?

[Nathan Goldbaum]

On Wed, Apr 27, 2016 at 8:49 PM, Alexander Griffing 
wrote:

> On Wed, Apr 27, 2016 at 7:48 PM, Elliot Hallmark 
> wrote:
> > Hello,
> >
> > I haven't worked hard yet to create a minimal runnable (reproduce-able)
> > example but I wanted to check if this sounds familiar to anyone.
> >
> > I have a pretty involved program that resizes arrays in place with
> > arr.resize.  When I run it with python it completes and gives the
> expected
> > result.  When I run it in Ipython, I get the following error:
> >
> > ```
> > ---> 43 self._buffer.resize((count,)+self._dim)
> >  44
> >  45
> >
> > ValueError: cannot resize an array that references or is referenced
> > by another array in this way.  Use the resize function
> > ```
> >
> > It was consistently doing this on the same array, after resizing two
> others
> > before hand, even after rebooting.  But trying to track it down it goes
> away
> > even if I undo everything I did to try and track it down.
>

You might find the %xdel magic to be useful:

In [1]: %xdel?
Docstring:
Delete a variable, trying to clear it from anywhere that
IPython's machinery has references to it. By default, this uses
the identity of the named object in the user namespace to remove
references held under other names. The object is also removed
from the output history.

Options
  -n : Delete the specified name from all namespaces, without
  checking their identity.



> >
> > Does this sound familiar?
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> I'd guess the issue is that your environment (IPython or IDLE or
> whatever) is keeping a reference to your array, for example so that
> you can refer to earlier outputs using the underscore _.  Here's how I
> was able to reproduce the problem:
>
> This is OK:
>
> >>> import numpy as np
> >>> a = np.arange(10)
> >>> a.resize(3)
> >>> a
> array([0, 1, 2])
>
> This gives an error:
>
> >>> import numpy as np
> >>> a = np.arange(10)
> >>> a
> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
> >>> a.resize(3)
> Traceback (most recent call last):
>   File "", line 1, in 
> ValueError: cannot resize an array that references or is referenced
> by another array in this way.  Use the resize function
>
> Cheers,
> Alex
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is this a known error in Ipython with numpy?

[Alexander Griffing]

On Wed, Apr 27, 2016 at 7:48 PM, Elliot Hallmark  wrote:
> Hello,
>
> I haven't worked hard yet to create a minimal runnable (reproduce-able)
> example but I wanted to check if this sounds familiar to anyone.
>
> I have a pretty involved program that resizes arrays in place with
> arr.resize.  When I run it with python it completes and gives the expected
> result.  When I run it in Ipython, I get the following error:
>
> ```
> ---> 43 self._buffer.resize((count,)+self._dim)
>  44
>  45
>
> ValueError: cannot resize an array that references or is referenced
> by another array in this way.  Use the resize function
> ```
>
> It was consistently doing this on the same array, after resizing two others
> before hand, even after rebooting.  But trying to track it down it goes away
> even if I undo everything I did to try and track it down.
>
> Does this sound familiar?
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>

I'd guess the issue is that your environment (IPython or IDLE or
whatever) is keeping a reference to your array, for example so that
you can refer to earlier outputs using the underscore _.  Here's how I
was able to reproduce the problem:

This is OK:

>>> import numpy as np
>>> a = np.arange(10)
>>> a.resize(3)
>>> a
array([0, 1, 2])

This gives an error:

>>> import numpy as np
>>> a = np.arange(10)
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> a.resize(3)
Traceback (most recent call last):
  File "", line 1, in 
ValueError: cannot resize an array that references or is referenced
by another array in this way.  Use the resize function

Cheers,
Alex
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Is this a known error in Ipython with numpy?

[Elliot Hallmark]

Hello,

I haven't worked hard yet to create a minimal runnable (reproduce-able)
example but I wanted to check if this sounds familiar to anyone.

I have a pretty involved program that resizes arrays in place with
arr.resize.  When I run it with python it completes and gives the expected
result.  When I run it in Ipython, I get the following error:

```
---> 43 self._buffer.resize((count,)+self._dim)
 44
 45

ValueError: cannot resize an array that references or is referenced
by another array in this way.  Use the resize function
```

It was consistently doing this on the same array, after resizing two others
before hand, even after rebooting.  But trying to track it down it goes
away even if I undo everything I did to try and track it down.

Does this sound familiar?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] nan version of einsum

[Chris Barber]

Hi,

Looks like I was a little confused.  It appears that the nan* versions of
functions in numpy just substitute the NaNs in a copy of the original array
and so are just convenience methods.  I was imagining that they were
optimized and handling the NaNs at a lower level.  It looks like the
"bottleneck" package tries to do this for nansum, nanprod, etc, but I don't
know if it is able to take advantage of SSE or not.

Anyways, maybe if I elaborate someone can offer suggestions.

My application is: given a large N-dimensional array, and a selected
(N-1)-dimensional slice of it, compute the Pearson correlation of that
slice versus all other (N-1)-dimensional slices of the array, omitting
NaN's from the calculation.

Ignoring NaN's for a moment, here is the slow and obvious way to do it:

from scipy.stats import pearsonr
import numpy as np
def corrs(data,index,dim):
seed = data.take(index,dim)
res = np.zeros(data.shape[dim])
for i in range(0,data.shape[dim]):
res[i] = pearsonr(seed, data.take(i,dim))[0]
return res

Doing all the math by hand and using einsum, there is an extremely fast
(though fairly cryptic) way of doing this:

def corrs2(data, index, axis):
seed = data.take([index], axis=axis)

sdims = range(0,seed.ndim)
ddims = range(0,data.ndim)
sample_axes = np.array([i for i in ddims if i != axis])

seed_mean = np.einsum(seed, sdims, []) / seed.size
data_mean = np.einsum(data, ddims, [axis]) / seed.size
data_mean.shape = tuple(data.shape[i] if i == axis else 1 for i in
ddims) # restore dims after einsum

seed_dev = np.einsum(seed-seed_mean, sdims, sdims)
numerator = np.einsum(seed_dev, ddims, data, ddims, [axis])
numerator -= np.einsum(seed_dev, ddims, data_mean, ddims, [axis])

denominator = np.einsum(data, ddims, data, ddims, [axis])
denominator += -2.0*np.einsum(data, ddims, data_mean, ddims, [axis])
denominator += np.sum(data_mean**2, axis=sample_axes) * seed.size
denominator *= np.einsum(seed_dev**2, sdims, [axis])
denominator = np.sqrt(denominator)

return np.clip(numerator / denominator, -1.0, 1.0)

It also doesn't need to make a copy of the array.  Re-introducing the
requirement to handle NaNs, though, I couldn't find any option besides
making a mask array and introducing that explicitly into the calculations.
That's why I was imagining an optimized "naneinsum."

Are there any existing ways of doing sums and products on numpy arrays that
have a fast way of handling NaNs?  Or is a mask array the best thing to
hope for?  I think that for this problem that I could transpose and reshape
the N-dimensional array down to 2-dimensions without making an array copy,
which might make it easier to interface with some optimizing package that
doesn't support multidimensional arrays fully.

I am fairly new to making numpy/python fast, and coming from MATLAB am very
impressed, though there are a bewlidering number of options when it comes
to trying to optimize.

Thanks,
Chris

On Tue, Apr 19, 2016 at 11:12 AM, Chris Barber 
wrote:

> Is there any interest in a nan-ignoring version of einsum a la nansum,
> nanprod, etc?  Any idea how difficult it would be to implement?
>
> - Chris
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] (no subject)

[Sebastian Berg]

On Mi, 2016-04-27 at 22:11 +0530, Saumyajit Dey wrote:
> Hi,
> 
> Thanks a lot for the reply. I am looking into the documentation
> already. Also is there any guide as to how the source code of Numpy
> is organised?
> 
> For example, when i write 
> 
> > np.power(2,3) 
> what is the workflow in terms of functions in different modules being
> called?
> 

No, there is not much. There are different paths/possibilities,
sometimes even intermingled. These are some:

1. Pure python functions, e.g. np.stack, np.delete, ...
   They are not hard to find/figure out (though some details may be),
   frankly, I usually just use the ?? magic in ipython.
2. Python shims for attributes, attributes usually go to the
   methods.c in numpy/core/src/multiarray
3. General C-Functions (Reshape, etc.) are usually in some specialized
   file in numpy/core/src/multiarray but wrapped by
   multiarraymodule.c, so you can backtrace from there. The exact calls
   can get pretty complex (even call back to python).
4. One important category (also tricky) are ufuncs. They form their own
   building block in the code base. The whole interplay of things is
   quite complex, so unless you need something specific I would be
   happy to understand all the things they can do for you and that
   they wrap C-functions working on a single axis in some sense.
   (code in numpy/core/src/umath)

Frankly, I often just "git grep ..." to find the right place to look
for something. It is not impossible to understand the logic behind the
files and what calls what, but I would claim it is usually faster to
grep for it if you are interested in something specific. There are some
more arcane things, such as code generations, but that is easier to ask
for/figure out for a specific problem. Things such as what happens when
a ufunc is called, and how the ufunc is created in the first place are
non-trivial.

NumPy has a few rather distinct building blocks. Ufuncs, the iterator,
general C-based shape/container functions, general python functions,
linear algebra, fft, polynoms, 

I would argue that finding something that interests you and trying to
figure that out and asking us about it explicitly is probably best.
Honestly, I think all of us devs have at least two things in the above
list we know almost nothing about. E.g. you don't need to understand
details of the FFT implementation unless you want to actually change
something there.

There are some "easy issues" marked in the git issue list, which may be
worth a shot if you like to just dive in. You could poke at one you
find interesting and then ask us (we might have tagged something as
"easy" but I would not guarantee all of them are, sometimes there are
unexpected difficulties or it is easy if you already know where to
look).

- Sebastian



> Regards,
> Saumyajit
> 
> Saumyajit Dey
> Junior Undergraduate Student:
> Department of Computer Science and Engineering
> National Institute of Technology
> Warangal (NITW), India
> Cell: +91-8885847028
> 
> On Wed, Apr 27, 2016 at 6:05 PM, Maniteja Nandana <
> maniteja.modesty...@gmail.com> wrote:
> > Hi,
> > Welcome! It would be a good exercise to look at the documentation
> > and tutorial for Numpy at http://docs.scipy.org/doc/
> > Also the lectures at the lectures at www.scipy-lectures.org might
> > be a interesting introduction to scientific python in numpy stack.
> > Hope it helps.
> > Happy learning !
> > Cheers,
> > Maniteja.
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> > 
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> > 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] (no subject)

[Saumyajit Dey]

Hi,

Thanks a lot for the reply. I am looking into the documentation already.
Also is there any guide as to how the source code of Numpy is organised?

For example, when i write

np.power(2,3)


​what is the workflow in terms of functions in different modules being
called?​

​Regards,
Saumyajit​

Saumyajit Dey
Junior Undergraduate Student:
Department of Computer Science and Engineering
National Institute of Technology
Warangal (NITW), India
Cell: +91-8885847028

On Wed, Apr 27, 2016 at 6:05 PM, Maniteja Nandana <
maniteja.modesty...@gmail.com> wrote:

> Hi,
>
> Welcome! It would be a good exercise to look at the documentation and
> tutorial for Numpy at http://docs.scipy.org/doc/
>
> Also the lectures at the lectures at www.scipy-lectures.org might be a
> interesting introduction to scientific python in numpy stack.
>
> Hope it helps.
>
> Happy learning !
>
> Cheers,
> Maniteja.
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Help with bit arrays

[Henrique Almeida]

 Hello, what's the current status on numpy for loading bit-arrays ?

I'm currently unable to correctly load black and white (1-bit) TIFF
images. Code example follows:

from PIL import Image
import numpy
from matplotlib import pyplot

img = Image.open('oi-00.tiff')
a = numpy.array(img)

^ does not work for 1-bit TIFF images

PIL source shows that it incorrectly uses typestr == '|b1'. I tried to
change this to '|t1', but I get :

TypeError: data type "|t1" not understood

My goal is to make the above code to work for black and white TIFF
images the same way it works for grayscale images. Any help ?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] (no subject)

[Maniteja Nandana]

Hi,

Welcome! It would be a good exercise to look at the documentation and
tutorial for Numpy at http://docs.scipy.org/doc/

Also the lectures at the lectures at www.scipy-lectures.org might be a
interesting introduction to scientific python in numpy stack.

Hope it helps.

Happy learning !

Cheers,
Maniteja.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion