[Numpy-discussion] LA improvements (was: dot function or dot notation, matrices, arrays?)

2009-12-22 Thread David Goldsmith
Starting a new thread for this.

On Tue, Dec 22, 2009 at 7:13 PM, Anne Archibald
 wrote:

> I think we have one major lacuna: vectorized linear algebra. If I have
> to solve a whole whack of four-dimensional linear systems, right now I
> need to either write a python loop and use linear algebra on them one
> by one, or implement my own linear algebra. It's a frustrating lacuna,
> because all the machinery is there: generalized ufuncs and LAPACK
> wrappers. Somebody just needs to glue them together. I've even tried
> making a start on it, but numpy's ufunc machinery and generic type
> system is just too much of a pain for me to make any progress as is.

Please be more specific: what (which aspects) have been "too much of a
pain"?  (I ask out of ignorance, not out of challenging your
opinion/experience.)

> I think if someone wanted to start building a low-level

Again, please be more specific: what do you mean by this?  (I know
generally what is meant by "low level," but I'd like you to spell out
a little more fully what you mean by this in this context.)

> generalized
> ufunc library interface to LAPACK, that would be a big improvement in
> numpy/scipy's linear algebra. Pretty much everything else strikes me
> as a question of notation. (Not to trivialize it: good notation makes
> a tremendous difference.)

Thanks, Anne.

DG

>
> Anne
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread David Cournapeau
On Wed, Dec 23, 2009 at 4:40 AM, Pauli Virtanen  wrote:

> I suppose raising an exception requires ownership of GIL.

I am curious: how did you know it was related to the GIL ? When I
tried debugging the issue, I could not tell whereas this was a problem
with python 2.7 or with numpy, and did not suspect the GIL at all.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread David Cournapeau
On Wed, Dec 23, 2009 at 12:50 AM, Bruce Southey  wrote:

> This still crashes Python 2.7 with the test_multiarray.TestIO.test_ascii.

Could you file a ticket next time ? I could not follow closely the
discussion the last week or so, and although I saw the crash, I missed
it was discussed already.

thanks,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] dot function or dot notation, matrices, arrays?

2009-12-22 Thread Anne Archibald
2009/12/21 David Goldsmith :
> On Mon, Dec 21, 2009 at 9:57 AM, Christopher Barker
>  wrote:
>> Dag Sverre Seljebotn wrote:
>>> I recently got motivated to get better linear algebra for Python;
>>
>> wonderful!
>>
>>> To me that seems like the ideal way to split up code -- let NumPy/SciPy
>>> deal with the array-oriented world and Sage the closer-to-mathematics
>>> notation.
>>
>> well, maybe -- but there is a lot of call for pure-computational linear
>> algebra. I do hope you'll consider building the computational portion of
>> it in a way that might be included in numpy or scipy by itself in the
>> future.
>
> My personal opinion is that the LA status quo is acceptably good:
> there's maybe a bit of an adjustment to make for newbies, but I don't
> see it as a very big one, and this list strikes me as very efficient
> at getting people over little bumps (e.g., someone emails in: "how do
> you matrix-multiply two arrays?" within minutes (:-)) Robert or
> Charles replies with "np.dot: np.dot([[1,2],[3,4]],[[1,2],[3,4]]) =
> array([[7,10],[15,22]])").  Certainly any significant changes to the
> base should need to run the gauntlet of an NEP process.

I think we have one major lacuna: vectorized linear algebra. If I have
to solve a whole whack of four-dimensional linear systems, right now I
need to either write a python loop and use linear algebra on them one
by one, or implement my own linear algebra. It's a frustrating lacuna,
because all the machinery is there: generalized ufuncs and LAPACK
wrappers. Somebody just needs to glue them together. I've even tried
making a start on it, but numpy's ufunc machinery and generic type
system is just too much of a pain for me to make any progress as is.

I think if someone wanted to start building a low-level generalized
ufunc library interface to LAPACK, that would be a big improvement in
numpy/scipy's linear algebra. Pretty much everything else strikes me
as a question of notation. (Not to trivialize it: good notation makes
a tremendous difference.)

Anne
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] dot function or dot notation, matrices, arrays?

2009-12-22 Thread David Goldsmith
On Tue, Dec 22, 2009 at 1:06 AM, Dag Sverre Seljebotn
 wrote:
>
> OK. As a digression, I think it is easy to get the wrong impression of
> Sage that it is for "symbolics" vs. "computations". The reality is that
> the symbolics has been one of the *weaker* aspects of Sage (though
> steadily improving) -- the strong aspect is computations, but with
> elements that NumPy doesn't handle efficiently: Arbitrary size integer
> and rationals, polynomials (or vectors of their coefficients if you wish
> -- just numbers, not symbols), and so on.
>
> So the Sage design is very much about computation, it is just that the
> standard floating point hasn't got all that much attention.
>

Good to know, Dag, thanks for the "digression." :-)

DG
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread Pauli Virtanen
ti, 2009-12-22 kello 15:28 -0700, Charles R Harris kirjoitti: [clip]
> But what about the GIL? That's what I'm curious about. Do we need to
> hold the GIL to check and clear and error? If so, there are other
> places where this will matter. I was under the impression that each
> thread had it's own error stack. But I don't know much about the GIL.

The issue seems to be that

Py_BEGIN_ALLOW_THREADS / NPY_BEGIN_ALLOW_THREADS

calls Python/ceval.c:PyEval_SaveThread(), which calls
Python/pystate.c:PyThreadState_Swap, which sets the current thread state
(Python/pystate.c:_PyThreadState_Current) to NULL.

I'm not 100% sure if this is the same thing as releasing GIL, GIL is
probably a subset of this.

But, the exception information lives in the thread state -> NULL pointer
dereference in PyErr_* -> BOOM. 
And yes,

PyObject *
PyErr_Occurred(void)
{
PyThreadState *tstate = PyThreadState_GET();

return tstate->curexc_type;
}

which probably means it shouldn't be called between ALLOW_THREADS. Needs
to be wrapped between NPY_ALLOW_C_API & NPY_DISABLE_C_API, which call
PyGILState_Ensure, which resurrects the thread state from some global
dictionary or something.

Pauli



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread Charles R Harris
On Tue, Dec 22, 2009 at 2:42 PM, Pauli Virtanen  wrote:

> ti, 2009-12-22 kello 14:32 -0700, Charles R Harris kirjoitti:
> [clip]
> > Could you expand a bit on this? There are several places where
> > PyErr_Occurred are called and I am wondering if there is a problem. In
> > fact, I moved one such check and a segfault went away, which made me
> > suspicious...
>
> I think here the point is that since PyOS_acii_strtod used to fail
> silently, also its replacement should -- so we'd need to clear any
> raised error before continuing.
>
>
But what about the GIL? That's what I'm curious about. Do we need to hold
the GIL to check and clear and error? If so, there are other places where
this will matter. I was under the impression that each thread had it's own
error stack. But I don't know much about the GIL.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread Pauli Virtanen
ti, 2009-12-22 kello 14:32 -0700, Charles R Harris kirjoitti:
[clip]
> Could you expand a bit on this? There are several places where
> PyErr_Occurred are called and I am wondering if there is a problem. In
> fact, I moved one such check and a segfault went away, which made me
> suspicious...

I think here the point is that since PyOS_acii_strtod used to fail
silently, also its replacement should -- so we'd need to clear any
raised error before continuing.

Pauli


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread Charles R Harris
On Tue, Dec 22, 2009 at 12:40 PM, Pauli Virtanen  wrote:

> ti, 2009-12-22 kello 10:16 -0700, Charles R Harris kirjoitti:
> [clip: PyOS_ascii_strtod -> PyOS_string_to_double]
> > The patch looks ok, but the functions handle errors differently and I
> > wonder if that has been completely audited.
>
> It can actually still crash from the same reason: PyOS_string_to_double
> docs say:
>
> """If no initial segment of the string is the valid representation of a
> floating-point number, set *endptr to point to the beginning of the
> string, raise ValueError, and return -1.0"""
>
> Indeed,
>
> $ gdb --args python3 -c "import numpy as np; np.fromstring('1,,', sep=',')"
> (gdb) run
> Program received signal SIGSEGV, Segmentation fault.
> PyErr_SetObject (exception=0x8291740, value=0xb7e926a0)
>at ../Python/errors.c:67
> 67  ../Python/errors.c: Tiedostoa tai hakemistoa ei ole.
>in ../Python/errors.c
> (gdb) bt
> #0  PyErr_SetObject (exception=0x8291740, value=0xb7e926a0)
>at ../Python/errors.c:67
> #1  0x080e8d5a in PyErr_Format (exception=0x8291740,
>format=0x81a0998 "could not convert string to float: %.200s")
>at ../Python/errors.c:638
> #2  0x080fb5fe in PyOS_string_to_double (s=0xb7ca2ae2 ",",
> endptr=0xbfffd130,
>overflow_exception=0x0) at ../Python/pystrtod.c:354
> #3  0x004a9bfc in NumPyOS_ascii_strtod (s=0xb7ca2ae2 ",",
> endptr=0xbfffd130)
>at numpy/core/src/multiarray/numpyos.c:525
>
> I suppose raising an exception requires ownership of GIL. So either we
> implement ASCII number parsing ourselves from scratch (or steal it from
> somewhere), or surround the call with appropriate GIL-acquiring wrappers
> plus if (PyErr_Occurred()) PyErr_Clear();
>
>
Could you expand a bit on this? There are several places where
PyErr_Occurred are called and I am wondering if there is a problem. In fact,
I moved one such check and a segfault went away, which made me suspicious...

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread Pauli Virtanen
ti, 2009-12-22 kello 10:16 -0700, Charles R Harris kirjoitti:
[clip: PyOS_ascii_strtod -> PyOS_string_to_double]
> The patch looks ok, but the functions handle errors differently and I
> wonder if that has been completely audited.

It can actually still crash from the same reason: PyOS_string_to_double
docs say:

"""If no initial segment of the string is the valid representation of a
floating-point number, set *endptr to point to the beginning of the
string, raise ValueError, and return -1.0"""

Indeed,

$ gdb --args python3 -c "import numpy as np; np.fromstring('1,,', sep=',')"
(gdb) run
Program received signal SIGSEGV, Segmentation fault.
PyErr_SetObject (exception=0x8291740, value=0xb7e926a0)
at ../Python/errors.c:67
67  ../Python/errors.c: Tiedostoa tai hakemistoa ei ole.
in ../Python/errors.c
(gdb) bt
#0  PyErr_SetObject (exception=0x8291740, value=0xb7e926a0)
at ../Python/errors.c:67
#1  0x080e8d5a in PyErr_Format (exception=0x8291740, 
format=0x81a0998 "could not convert string to float: %.200s")
at ../Python/errors.c:638
#2  0x080fb5fe in PyOS_string_to_double (s=0xb7ca2ae2 ",",
endptr=0xbfffd130, 
overflow_exception=0x0) at ../Python/pystrtod.c:354
#3  0x004a9bfc in NumPyOS_ascii_strtod (s=0xb7ca2ae2 ",",
endptr=0xbfffd130)
at numpy/core/src/multiarray/numpyos.c:525

I suppose raising an exception requires ownership of GIL. So either we
implement ASCII number parsing ourselves from scratch (or steal it from
somewhere), or surround the call with appropriate GIL-acquiring wrappers
plus if (PyErr_Occurred()) PyErr_Clear();

Anyway, how malformed input is handled is currently not so well
specified anyway, so this part requires further fixes.

-- 
Pauli Virtanen


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread Charles R Harris
On Tue, Dec 22, 2009 at 8:50 AM, Bruce Southey  wrote:

> On 12/22/2009 09:05 AM, David Cournapeau wrote:
>
>> Hi,
>>
>> I have just released the 2nd release candidate for numpy 1.4.0, which
>> fixes a few critical bugs founds since the RC1. Tarballs and binary
>> installers for numpy/scipy may be found on
>> https://sourceforge.net/projects/numpy.
>>
>> cheers,
>>
>> David
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> Hi,
> This still crashes Python 2.7 with the test_multiarray.TestIO.test_ascii.
>
> The file numpy/core/src/multiarray/numpyos.c needs a change as per this
> thread:
> "test_multiarray.TestIO.test_ascii segmentationfault with Python2.7"
> http://mail.scipy.org/pipermail/numpy-discussion/2009-December/047481.html
>
> The segmentation fault is avoided with this patch derived from the current
> numpy-1.4RC2 version and not the SVN version.
>
>
The patch looks ok, but the functions handle errors differently and I wonder
if that has been completely audited.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread Gökhan Sever
On Tue, Dec 22, 2009 at 9:05 AM, David Cournapeau wrote:

> Hi,
>
> I have just released the 2nd release candidate for numpy 1.4.0, which
> fixes a few critical bugs founds since the RC1. Tarballs and binary
> installers for numpy/scipy may be found on
> https://sourceforge.net/projects/numpy.
>
> cheers,
>
> David
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

This release results with the same import error on my system that I posted
on
http://old.nabble.com/Another-numpy-svn-installation-error-td26878029.html

[gse...@ccn Desktop]$ python
Python 2.6 (r26:66714, Jun  8 2009, 16:07:26)
[GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/gsever/Desktop/python-repo/numpy/numpy/__init__.py", line 132,
in 
import add_newdocs
  File "/home/gsever/Desktop/python-repo/numpy/numpy/add_newdocs.py", line
9, in 
from lib import add_newdoc
  File "/home/gsever/Desktop/python-repo/numpy/numpy/lib/__init__.py", line
4, in 
from type_check import *
  File "/home/gsever/Desktop/python-repo/numpy/numpy/lib/type_check.py",
line 8, in 
import numpy.core.numeric as _nx
  File "/home/gsever/Desktop/python-repo/numpy/numpy/core/__init__.py", line
6, in 
import umath
ImportError: /home/gsever/Desktop/python-repo/numpy/numpy/core/umath.so:
undefined symbol: npy_spacing

Is there any remedy for this error?


-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Proposal for matrix_rank function in numpy

2009-12-22 Thread Matthew Brett
Hi,

> I'm happy to write the doctests as tests.   My feeling is there is no
> objection to this function at the moment, so it would be reasonable,
> unless I hear otherwise, to commit to SVN.

Committed - with tests in tests_linalg.py - in revision 8029

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread Bruce Southey

On 12/22/2009 09:05 AM, David Cournapeau wrote:

Hi,

I have just released the 2nd release candidate for numpy 1.4.0, which
fixes a few critical bugs founds since the RC1. Tarballs and binary
installers for numpy/scipy may be found on
https://sourceforge.net/projects/numpy.

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
   

Hi,
This still crashes Python 2.7 with the test_multiarray.TestIO.test_ascii.

The file numpy/core/src/multiarray/numpyos.c needs a change as per this 
thread:

"test_multiarray.TestIO.test_ascii segmentationfault with Python2.7"
http://mail.scipy.org/pipermail/numpy-discussion/2009-December/047481.html

The segmentation fault is avoided with this patch derived from the 
current numpy-1.4RC2 version and not the SVN version.


Bruce


--- numpyos.c   2009-12-22 09:36:58.0 -0600
+++ ../numpy-1.4.0rc2/numpy/core/src/multiarray/numpyos.c   2009-12-22 
09:44:04.0 -0600
@@ -506,7 +506,11 @@
 }
 memcpy(buffer, s, n);
 buffer[n] = '\0';
+#if PY_VERSION_HEX >= 0x0207
+result = PyOS_string_to_double(buffer, &q, NULL);
+#else
 result = PyOS_ascii_strtod(buffer, &q);
+#endif
 if (endptr != NULL) {
 *endptr = (char*)(s + (q - buffer));
 }
@@ -515,7 +519,11 @@
 }
 /* End of ##2 */
 
+#if PY_VERSION_HEX >= 0x0207
+return PyOS_string_to_double(s, endptr, NULL);
+#else
 return PyOS_ascii_strtod(s, endptr);
+#endif
 }
 
 
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


[Numpy-discussion] [ANN] numpy 1.4.0 rc2

2009-12-22 Thread David Cournapeau
Hi,

I have just released the 2nd release candidate for numpy 1.4.0, which
fixes a few critical bugs founds since the RC1. Tarballs and binary
installers for numpy/scipy may be found on
https://sourceforge.net/projects/numpy.

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


[Numpy-discussion] ANN: PyTables 2.2b2 released

2009-12-22 Thread Francesc Alted
===
 Announcing PyTables 2.2b2
===

PyTables is a library for managing hierarchical datasets and designed to
efficiently cope with extremely large amounts of data with support for
full 64-bit file addressing.  PyTables runs on top of the HDF5 library
and NumPy package for achieving maximum throughput and convenient use.

This is the second beta version of 2.2 release.  The main addition is
the support for links.  All HDF5 kind of links are supported: hard, soft
and external.  Hard and soft links are similar to hard and symbolic
links in regular UNIX filesystems, while external links are more like
mounting external filesystems (in this case, HDF5 files) on top of
existing ones.  This allows for a considerable degree of flexibility
when defining your object tree.  See the new tutorial at:

http://www.pytables.org/docs/manual-2.2b2/ch03.html#LinksTutorial

Also, some other new features (like complete control of HDF5 chunk cache
parameters and native compound types in attributes), bug fixes and a
couple of (small) API changes happened.

In case you want to know more in detail what has changed in this
version, have a look at:

http://www.pytables.org/moin/ReleaseNotes/Release_2.2b2

You can download a source package with generated PDF and HTML docs, as
well as binaries for Windows, from:

http://www.pytables.org/download/preliminary

For an on-line version of the manual, visit:

http://www.pytables.org/docs/manual-2.2b2


Resources
=

About PyTables:

http://www.pytables.org

About the HDF5 library:

http://hdfgroup.org/HDF5/

About NumPy:

http://numpy.scipy.org/


Acknowledgments
===

Thanks to many users who provided feature improvements, patches, bug
reports, support and suggestions.  See the ``THANKS`` file in the
distribution package for a (incomplete) list of contributors.  Most
specially, a lot of kudos go to the HDF5 and NumPy (and numarray!)
makers.  Without them, PyTables simply would not exist.


Share your experience
=

Let us know of any bugs, suggestions, gripes, kudos, etc. you may
have.




  **Enjoy data!**

  -- The PyTables Team

-- 
Francesc Alted
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] dot function or dot notation, matrices, arrays?

2009-12-22 Thread Dag Sverre Seljebotn
Christopher Barker wrote:
> Dag Sverre Seljebotn wrote:
>> This is readily done -- there is no computational portion except for 
>> what is in NumPy/Scipy or scikits, and I intend for it to remain that 
>> way. It's just another interface, really.
>>
>> (What kind of computations were you thinking about?)
> 
> Nothing in particular -- just computational as opposed to symbolic 
> manipulation.

OK. As a digression, I think it is easy to get the wrong impression of 
Sage that it is for "symbolics" vs. "computations". The reality is that 
the symbolics has been one of the *weaker* aspects of Sage (though 
steadily improving) -- the strong aspect is computations, but with 
elements that NumPy doesn't handle efficiently: Arbitrary size integer 
and rationals, polynomials (or vectors of their coefficients if you wish 
-- just numbers, not symbols), and so on.

So the Sage design is very much about computation, it is just that the 
standard floating point hasn't got all that much attention.

-- 
Dag Sverre
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion