Re: [Numpy-discussion] storage order and concatenate
Travis Oliphant wrote: > David Cournapeau wrote: > >> Hi, >> >> >> What are the rules concerning storage with numpy ? >> > The rule is that a numpy array has "strides" which specify how many > bytes to skip to get to the next element in the array. That's the > internal model. There are no hard and fast rules about storage order. > Internally, C-order is as good as Fortran-order (except the array > iterator gives special preference to C-order and all functions for which > the order can be specified (like zeros) default to C-order). > Ok, this is again a bad habit from matlab to think in C or F order... > Thus, the storage order is whatever the strides say it is. Now, there > are flags that keep track of whether or not the strides agree with the 2 > recognized special cases of "Fortran-order" (first-index varies the > fastest) or "C-order" (last-index varies the fastest). But, this is > only for convenience. Very few functions actually require a > specification of storage order. Those that allow it default to "C-order". > > You can't think of a NumPy array has having a particular storage order > unless you explicitly request it. One of the most common ways that > Fortran-order arrays show up, for example is when a C-order array is > transposed. A transpose operation does nothing except flip the strides > (and therefore the flags) of the array. This is what is happening in > concatenate (using axis=1) to give you a Fortran-order array. > Bascially, code equivalent to the following is being run: > concatenate([X1.T, X2.T]).T > > In the second example, you explicitly create the array (and therefore > the strides) as C-order and then fill it (so it doesn't change on you). > The first example used array calculations which don't guarantee the > storage order. > > This is all seamless to the user until you have to interface with > extension code. Ideally, you write compiled code that deals with > strided arrays. If you can't, then you request an array of the required > storage-order. > > By the way, for interfacing with ctypes, check out the > ctypeslib.ndpointer class-creation function for flag checking and the > require function for automatic conversion of an array to specific > requirements. > > I tried to to that at first, but I couldn't make the examples of numpy works; after having tried at home with beta versions, it looks like it is a ctype version problem, as it works with ctype 1.0 + numpy 1.0rc1. Thanks for the explanations, this answers most questions I had in mind for numpy internal layout compared to matlab which I am used to. I think this should be in the wiki somewhere; do you mind if I use your email as a basis for the tentative numpy tutorial (memory layout section, maybe) ? David - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Negative values with unsigned data types problems
Hi,
I'm writing this here because the numpy Trac seems down:
{{{
Oops...
Trac detected an internal error: The Trac Environment needs to be upgraded.
Run trac-admin /home/scipy/trac/numpy upgrade"
}}}
Well, it seems that there are inconsistencies on creating arrays coming from
negative values using unsigned integers:
In [41]: numpy.asarray([-1], dtype='uint8')
Out[41]: array([255], dtype=uint8)
In [42]: numpy.asarray([-1], dtype='uint16')
Out[42]: array([65535], dtype=uint16)
until here it's fine, but:
In [43]: numpy.asarray([-1], dtype='uint32')
---
Traceback (most recent call last)
/home/faltet/python.nobackup/numpy/ in ()
/usr/local/lib/python2.5/site-packages/numpy/core/numeric.py in asarray(a,
dtype, order)
129 are converted to base class ndarray.
130 """
--> 131 return array(a, dtype, copy=False, order=order)
132
133 def asanyarray(a, dtype=None, order=None):
: long() argument must be a string or a number,
not 'list'
and the same happens with 'uint64'.
My numpy version is 1.0.dev3216
Regards,
--
>0,0< Francesc Altet http://www.carabos.com/
V V Cárabos Coop. V. Enjoy Data
"-"
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Incorrect removal of NULL char in buffers
Hi,
I'm trying to build-up numpy arrays coming from buffers, and I'm getting a
somewhat unexpected result.
First, for numeric values, everything seems ok (i.e. the NULL character is
correctly interpretated), and works equally for both numarray and numpy:
In [98]: numarray.array("a\x00b"*4, dtype='Float32',shape=3)
Out[98]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20],
type=Float32)
In [99]: numpy.ndarray(buffer="a\x00b"*4, dtype='Float32',shape=3)
Out[99]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20],
dtype=float32)
However, for string values, numpy seems to work in a strange way.
The numarray have an expected behaviour, IMO:
In [100]: numarray.strings.array(buffer="a\x00b"*4, itemsize=4, shape=3)
Out[100]: CharArray(['a', '', 'ba'])
but numpy haven't:
In [101]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3)
Out[101]:
array([aba, ba, bab],
dtype='|S4')
i.e. it seems like numpy is striping-off NULL chars before building the object
and I don't think this is correct.
Cheers,
--
>0,0< Francesc Altet http://www.carabos.com/
V V Cárabos Coop. V. Enjoy Data
"-"
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Non-writeable default for numpy.ndarray
Hello, Is the next a bug a feature? In [102]: f4=numpy.ndarray(buffer="a\x00b"*4, dtype="f4", shape=3) In [103]: f4 Out[103]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], dtype=float32) In [104]: f4[2] = 2 --- Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ in () : array is not writeable In [105]: f4.flags.writeable = True In [106]: f4[2] = 2 In [107]: f4 Out[107]: array([ 2.60561966e+20, 8.94319890e-39, 2.e+00], dtype=float32) i.e. in an array built from ndarray, the default is that it has to be read-only? -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-" - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-writeable default for numpy.ndarray
Francesc Altet wrote: > Hello, > > Is the next a bug a feature? > > In [102]: f4=numpy.ndarray(buffer="a\x00b"*4, dtype="f4", shape=3) > > In [103]: f4 > Out[103]: array([ 2.60561966e+20, 8.94319890e-39, 5.92050103e+20], > dtype=float32) > > In [104]: f4[2] = 2 > --- > Traceback (most recent call last) > > /home/faltet/python.nobackup/numpy/ in () > > : array is not writeable > > In [105]: f4.flags.writeable = True > > In [106]: f4[2] = 2 > > In [107]: f4 > Out[107]: array([ 2.60561966e+20, 8.94319890e-39, 2.e+00], > dtype=float32) > > > i.e. in an array built from ndarray, the default is that it has to be > read-only? > It's not that the it's being built from ndarray, it's that the buffer that you are passing it is read only. In fact, I'd argue that allowing the writeable flag to be set to True in this case is actually a bug. Consider this slightly modified example: >>> a = "12345"*4 >>> f4=numpy.ndarray(buffer=a, dtype="f4", shape=3) >>> f4[2] = 99 Traceback (most recent call last): File "", line 1, in ? RuntimeError: array is not writeable >>> f4.flags.writeable = True >>> a '12345123451234512345' >>> f4.flags.writeable = True >>> f4[2] = 99 >>> a '12345123\x00\x00\xc6B34512345' The original, *immutable* string has been mutated. This could get you into real trouble in certain situations. -tim - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-writeable default for numpy.ndarray
A Divendres 29 Setembre 2006 18:12, Tim Hochberg va escriure: > It's not that the it's being built from ndarray, it's that the buffer > that you are passing it is read only. In fact, I'd argue that allowing > the writeable flag to be set to True in this case is actually a bug. > > Consider this slightly modified example: > >>> a = "12345"*4 > >>> f4=numpy.ndarray(buffer=a, dtype="f4", shape=3) > >>> f4[2] = 99 > > Traceback (most recent call last): > File "", line 1, in ? > RuntimeError: array is not writeable > > >>> f4.flags.writeable = True > >>> a > > '12345123451234512345' > > >>> f4.flags.writeable = True > >>> f4[2] = 99 > >>> a > > '12345123\x00\x00\xc6B34512345' > > The original, *immutable* string has been mutated. This could get you > into real trouble in certain situations. I see. Thanks for the explanation. -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-" - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-writeable default for numpy.ndarray
Tim Hochberg wrote: > Francesc Altet wrote: > > It's not that the it's being built from ndarray, it's that the buffer > that you are passing it is read only. This is correct. > In fact, I'd argue that allowing > the writeable flag to be set to True in this case is actually a bug. > It's actually intentional. Strings used as buffers are allowed to be writeable. This is an explicit design decision to allow pickles to load without making 2 copies of the memory. The pickled string that Python creates is used as the actual memory for loaded arrays. Now, I suppose it would be possible to still allow this but be more finnicky about when a string-used-as-the-memory can be set writeable (i.e. we are the only reference to it). But, this would be a fragile solution as well. My inclination is to just warn users not to use strings as buffers unless they know what they are doing. The fact that it is read-only by default is enough of a guard against "accidentally" altering a string you didn't want to alter. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-writeable default for numpy.ndarray
Travis Oliphant wrote: > Tim Hochberg wrote: > >> Francesc Altet wrote: >> >> It's not that the it's being built from ndarray, it's that the buffer >> that you are passing it is read only. >> > This is correct. > >> In fact, I'd argue that allowing >> the writeable flag to be set to True in this case is actually a bug. >> >> > It's actually intentional. Strings used as buffers are allowed to be > writeable. This is an explicit design decision to allow pickles to load > without making 2 copies of the memory. The pickled string that Python > creates is used as the actual memory for loaded arrays. > > Now, I suppose it would be possible to still allow this but be more > finnicky about when a string-used-as-the-memory can be set writeable > (i.e. we are the only reference to it). But, this would be a fragile > solution as well. > > My inclination is to just warn users not to use strings as buffers > unless they know what they are doing. The fact that it is read-only by > default is enough of a guard against "accidentally" altering a string > you didn't want to alter. > That makes sense. Someday, when string/unicode => bytes/string, we'll be able to use the bytes type for this and it'll be much cleaner. But not for a while I imagine. -tim - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Negative values with unsigned data types problems
Francesc Altet wrote:
> Hi,
>
> I'm writing this here because the numpy Trac seems down:
>
> {{{
> Oops...
>
> Trac detected an internal error: The Trac Environment needs to be upgraded.
> Run trac-admin /home/scipy/trac/numpy upgrade"
> }}}
It's back up.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Incorrect removal of NULL char in buffers
Francesc Altet wrote: > Hi, > > > However, for string values, numpy seems to work in a strange way. > The numarray have an expected behaviour, IMO: > > In [100]: numarray.strings.array(buffer="a\x00b"*4, itemsize=4, shape=3) > Out[100]: CharArray(['a', '', 'ba']) > > I'm not sure why you think this is "expected." You have non-terminating NULLs in this array and yet they are not printing for you. Just look at the tostring()... > but numpy haven't: > > In [101]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) > Out[101]: > array([aba, ba, bab], > dtype='|S4') > > i.e. it seems like numpy is striping-off NULL chars before building the > object > and I don't think this is correct. > Hmmm. I don't see that at all. This is what I get (version of numpy is 1.0.dev3233) In [33]: numpy.ndarray(buffer="a\x00b"*4, dtype="S4", shape=3) Out[33]: array(['a\x00ba', '\x00ba', 'ba\x00b'], dtype='|S4') which to me is very much expected. I.e. only terminating NULLs are stripped off of the strings on printing. I think you are getting different results because string printing used to not include the quotes (which had the side-effect of not printing NULLs in the middle of strings). They are still there, just not showing up in your output. In the end both numarray and numpy have the same data stored internally. It's just a matter of how it is being printed that seems to differ a bit. From my perspective, only NULLs at the end of strings should be stripped off and that is the (current) behavior of NumPy. You are getting different results, because the array-printing for strings was recently updated (to insert the quotes so that it makes more sense).Without these changes, I think the NULLs were being stripped away on printing. In other words, something like print 'a\x00ba' aba used to be happening. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Q: NumPy (svn) with cygwin build error?
Hello. Starting from a freshly installed cygwin, I have built Python 2.5, ATLAS and LAPACK all from source using gcc 3.4.4 (cygming special) for both the C and f77 compilers. Now I am trying to build NumPy from SVN. I have edited site.cfg so that python setup.py config --compiler=mingw32 finds all of the linear algebra packs and completes without error. Then I fire off python setup.py build --compiler=mingw32 and I soon get (while building extension numpy.random.mtrand) _configtest.c:7:2: #error No _WIN32 which is not immediately fatal but definitely a concern. Soon after, building numpy.core.multiarray gives me this: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall creating build/temp.cygwin-1.5.21-i686-2.5 creating build/temp.cygwin-1.5.21-i686-2.5/numpy creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src compile options: '-Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src/multiarraymodule.o In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory I am puzzzled, because /usr/include/sys/select.h does exist. But maybe that's not what pyport.h is actually looking for? I've been banging away at this for several hours, so I must be doing something silly. Any help will be much appreciated. Thanks! _ Express yourself - download free Windows Live Messenger themes! http://clk.atdmt.com/MSN/go/msnnkwme002001msn/direct/01/?href=http://imagine-msn.com/themes/vibe/default.aspx?locale=en-us&source=hmtagline - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-writeable default for numpy.ndarray
Francesc Altet wrote: >I see. Thanks for the explanation. > > You deserve the thanks for the great testing of less-traveled corners of NumPy. It's exactly the kind of thing needed to get NumPy ready for release. -Travis - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Problem building from svn under cygwin
I have the latest cygwin environment and am using gcc 3.4.4. to build numpy. Configure works: python setup.py configure --compiler=mingw32 Build fails: python setup.py build --compiler=mingw32 While building numpy.random.mtrand I get this: _configtest.c:7:2: #error No _WIN32 and a little bit later the build aborts completely on multiarray with this: building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -mno-cygwin -mdll -O -Wall creating build/temp.cygwin-1.5.21-i686-2.5 creating build/temp.cygwin-1.5.21-i686-2.5/numpy creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core creating build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src compile options: '-Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' gcc -mno-cygwin -mdll -O -Wall -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core/src -Inumpy/core/include -Ibuild/src.cygwin-1.5.21-i686-2.5/numpy/core -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c numpy/core/src/multiarraymodule.c -o build/temp.cygwin-1.5.21-i686-2.5/numpy/core/src/multiarraymodule.o In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory In file included from /usr/local/include/python2.5/Python.h:57, from numpy/core/src/multiarraymodule.c:18: /usr/local/include/python2.5/pyport.h:226:24: sys/select.h: No such file or directory I am puzzled, because /usr/include/sys/select.h actually does exist on my cygwin installation. But maybe that's not what pyport.h is actually looking for??? I've been banging away at this for several hours, so I must be doing something silly. Any help will be much appreciated. Thanks! - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem building from svn under cygwin
Coatimundi wrote: > I have the latest cygwin environment and am using gcc 3.4.4. to build numpy. > > Configure works: > > python setup.py configure --compiler=mingw32 > > Build fails: > > python setup.py build --compiler=mingw32 What Python are you using? It looks like you built Python 2.5 using cygwin (that is, the UNIX emulation layer). You cannot build your extensions with mingw32 in that case. Don't use --compiler at all and you will be fine. If you are using the regular Python binaries, then you should be using --compiler=mingw32, but you're getting the wrong Python headers in that case. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
