[Numpy-discussion] LA improvements (was: dot function or dot notation, matrices, arrays?)
Starting a new thread for this. On Tue, Dec 22, 2009 at 7:13 PM, Anne Archibald wrote: > I think we have one major lacuna: vectorized linear algebra. If I have > to solve a whole whack of four-dimensional linear systems, right now I > need to either write a python loop and use linear algebra on them one > by one, or implement my own linear algebra. It's a frustrating lacuna, > because all the machinery is there: generalized ufuncs and LAPACK > wrappers. Somebody just needs to glue them together. I've even tried > making a start on it, but numpy's ufunc machinery and generic type > system is just too much of a pain for me to make any progress as is. Please be more specific: what (which aspects) have been "too much of a pain"? (I ask out of ignorance, not out of challenging your opinion/experience.) > I think if someone wanted to start building a low-level Again, please be more specific: what do you mean by this? (I know generally what is meant by "low level," but I'd like you to spell out a little more fully what you mean by this in this context.) > generalized > ufunc library interface to LAPACK, that would be a big improvement in > numpy/scipy's linear algebra. Pretty much everything else strikes me > as a question of notation. (Not to trivialize it: good notation makes > a tremendous difference.) Thanks, Anne. DG > > Anne > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
On Wed, Dec 23, 2009 at 4:40 AM, Pauli Virtanen wrote: > I suppose raising an exception requires ownership of GIL. I am curious: how did you know it was related to the GIL ? When I tried debugging the issue, I could not tell whereas this was a problem with python 2.7 or with numpy, and did not suspect the GIL at all. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
On Wed, Dec 23, 2009 at 12:50 AM, Bruce Southey wrote: > This still crashes Python 2.7 with the test_multiarray.TestIO.test_ascii. Could you file a ticket next time ? I could not follow closely the discussion the last week or so, and although I saw the crash, I missed it was discussed already. thanks, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dot function or dot notation, matrices, arrays?
2009/12/21 David Goldsmith : > On Mon, Dec 21, 2009 at 9:57 AM, Christopher Barker > wrote: >> Dag Sverre Seljebotn wrote: >>> I recently got motivated to get better linear algebra for Python; >> >> wonderful! >> >>> To me that seems like the ideal way to split up code -- let NumPy/SciPy >>> deal with the array-oriented world and Sage the closer-to-mathematics >>> notation. >> >> well, maybe -- but there is a lot of call for pure-computational linear >> algebra. I do hope you'll consider building the computational portion of >> it in a way that might be included in numpy or scipy by itself in the >> future. > > My personal opinion is that the LA status quo is acceptably good: > there's maybe a bit of an adjustment to make for newbies, but I don't > see it as a very big one, and this list strikes me as very efficient > at getting people over little bumps (e.g., someone emails in: "how do > you matrix-multiply two arrays?" within minutes (:-)) Robert or > Charles replies with "np.dot: np.dot([[1,2],[3,4]],[[1,2],[3,4]]) = > array([[7,10],[15,22]])"). Certainly any significant changes to the > base should need to run the gauntlet of an NEP process. I think we have one major lacuna: vectorized linear algebra. If I have to solve a whole whack of four-dimensional linear systems, right now I need to either write a python loop and use linear algebra on them one by one, or implement my own linear algebra. It's a frustrating lacuna, because all the machinery is there: generalized ufuncs and LAPACK wrappers. Somebody just needs to glue them together. I've even tried making a start on it, but numpy's ufunc machinery and generic type system is just too much of a pain for me to make any progress as is. I think if someone wanted to start building a low-level generalized ufunc library interface to LAPACK, that would be a big improvement in numpy/scipy's linear algebra. Pretty much everything else strikes me as a question of notation. (Not to trivialize it: good notation makes a tremendous difference.) Anne ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dot function or dot notation, matrices, arrays?
On Tue, Dec 22, 2009 at 1:06 AM, Dag Sverre Seljebotn wrote: > > OK. As a digression, I think it is easy to get the wrong impression of > Sage that it is for "symbolics" vs. "computations". The reality is that > the symbolics has been one of the *weaker* aspects of Sage (though > steadily improving) -- the strong aspect is computations, but with > elements that NumPy doesn't handle efficiently: Arbitrary size integer > and rationals, polynomials (or vectors of their coefficients if you wish > -- just numbers, not symbols), and so on. > > So the Sage design is very much about computation, it is just that the > standard floating point hasn't got all that much attention. > Good to know, Dag, thanks for the "digression." :-) DG ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
ti, 2009-12-22 kello 15:28 -0700, Charles R Harris kirjoitti: [clip]
> But what about the GIL? That's what I'm curious about. Do we need to
> hold the GIL to check and clear and error? If so, there are other
> places where this will matter. I was under the impression that each
> thread had it's own error stack. But I don't know much about the GIL.
The issue seems to be that
Py_BEGIN_ALLOW_THREADS / NPY_BEGIN_ALLOW_THREADS
calls Python/ceval.c:PyEval_SaveThread(), which calls
Python/pystate.c:PyThreadState_Swap, which sets the current thread state
(Python/pystate.c:_PyThreadState_Current) to NULL.
I'm not 100% sure if this is the same thing as releasing GIL, GIL is
probably a subset of this.
But, the exception information lives in the thread state -> NULL pointer
dereference in PyErr_* -> BOOM.
And yes,
PyObject *
PyErr_Occurred(void)
{
PyThreadState *tstate = PyThreadState_GET();
return tstate->curexc_type;
}
which probably means it shouldn't be called between ALLOW_THREADS. Needs
to be wrapped between NPY_ALLOW_C_API & NPY_DISABLE_C_API, which call
PyGILState_Ensure, which resurrects the thread state from some global
dictionary or something.
Pauli
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
On Tue, Dec 22, 2009 at 2:42 PM, Pauli Virtanen wrote: > ti, 2009-12-22 kello 14:32 -0700, Charles R Harris kirjoitti: > [clip] > > Could you expand a bit on this? There are several places where > > PyErr_Occurred are called and I am wondering if there is a problem. In > > fact, I moved one such check and a segfault went away, which made me > > suspicious... > > I think here the point is that since PyOS_acii_strtod used to fail > silently, also its replacement should -- so we'd need to clear any > raised error before continuing. > > But what about the GIL? That's what I'm curious about. Do we need to hold the GIL to check and clear and error? If so, there are other places where this will matter. I was under the impression that each thread had it's own error stack. But I don't know much about the GIL. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
ti, 2009-12-22 kello 14:32 -0700, Charles R Harris kirjoitti: [clip] > Could you expand a bit on this? There are several places where > PyErr_Occurred are called and I am wondering if there is a problem. In > fact, I moved one such check and a segfault went away, which made me > suspicious... I think here the point is that since PyOS_acii_strtod used to fail silently, also its replacement should -- so we'd need to clear any raised error before continuing. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
On Tue, Dec 22, 2009 at 12:40 PM, Pauli Virtanen wrote:
> ti, 2009-12-22 kello 10:16 -0700, Charles R Harris kirjoitti:
> [clip: PyOS_ascii_strtod -> PyOS_string_to_double]
> > The patch looks ok, but the functions handle errors differently and I
> > wonder if that has been completely audited.
>
> It can actually still crash from the same reason: PyOS_string_to_double
> docs say:
>
> """If no initial segment of the string is the valid representation of a
> floating-point number, set *endptr to point to the beginning of the
> string, raise ValueError, and return -1.0"""
>
> Indeed,
>
> $ gdb --args python3 -c "import numpy as np; np.fromstring('1,,', sep=',')"
> (gdb) run
> Program received signal SIGSEGV, Segmentation fault.
> PyErr_SetObject (exception=0x8291740, value=0xb7e926a0)
>at ../Python/errors.c:67
> 67 ../Python/errors.c: Tiedostoa tai hakemistoa ei ole.
>in ../Python/errors.c
> (gdb) bt
> #0 PyErr_SetObject (exception=0x8291740, value=0xb7e926a0)
>at ../Python/errors.c:67
> #1 0x080e8d5a in PyErr_Format (exception=0x8291740,
>format=0x81a0998 "could not convert string to float: %.200s")
>at ../Python/errors.c:638
> #2 0x080fb5fe in PyOS_string_to_double (s=0xb7ca2ae2 ",",
> endptr=0xbfffd130,
>overflow_exception=0x0) at ../Python/pystrtod.c:354
> #3 0x004a9bfc in NumPyOS_ascii_strtod (s=0xb7ca2ae2 ",",
> endptr=0xbfffd130)
>at numpy/core/src/multiarray/numpyos.c:525
>
> I suppose raising an exception requires ownership of GIL. So either we
> implement ASCII number parsing ourselves from scratch (or steal it from
> somewhere), or surround the call with appropriate GIL-acquiring wrappers
> plus if (PyErr_Occurred()) PyErr_Clear();
>
>
Could you expand a bit on this? There are several places where
PyErr_Occurred are called and I am wondering if there is a problem. In fact,
I moved one such check and a segfault went away, which made me suspicious...
Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
ti, 2009-12-22 kello 10:16 -0700, Charles R Harris kirjoitti:
[clip: PyOS_ascii_strtod -> PyOS_string_to_double]
> The patch looks ok, but the functions handle errors differently and I
> wonder if that has been completely audited.
It can actually still crash from the same reason: PyOS_string_to_double
docs say:
"""If no initial segment of the string is the valid representation of a
floating-point number, set *endptr to point to the beginning of the
string, raise ValueError, and return -1.0"""
Indeed,
$ gdb --args python3 -c "import numpy as np; np.fromstring('1,,', sep=',')"
(gdb) run
Program received signal SIGSEGV, Segmentation fault.
PyErr_SetObject (exception=0x8291740, value=0xb7e926a0)
at ../Python/errors.c:67
67 ../Python/errors.c: Tiedostoa tai hakemistoa ei ole.
in ../Python/errors.c
(gdb) bt
#0 PyErr_SetObject (exception=0x8291740, value=0xb7e926a0)
at ../Python/errors.c:67
#1 0x080e8d5a in PyErr_Format (exception=0x8291740,
format=0x81a0998 "could not convert string to float: %.200s")
at ../Python/errors.c:638
#2 0x080fb5fe in PyOS_string_to_double (s=0xb7ca2ae2 ",",
endptr=0xbfffd130,
overflow_exception=0x0) at ../Python/pystrtod.c:354
#3 0x004a9bfc in NumPyOS_ascii_strtod (s=0xb7ca2ae2 ",",
endptr=0xbfffd130)
at numpy/core/src/multiarray/numpyos.c:525
I suppose raising an exception requires ownership of GIL. So either we
implement ASCII number parsing ourselves from scratch (or steal it from
somewhere), or surround the call with appropriate GIL-acquiring wrappers
plus if (PyErr_Occurred()) PyErr_Clear();
Anyway, how malformed input is handled is currently not so well
specified anyway, so this part requires further fixes.
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
On Tue, Dec 22, 2009 at 8:50 AM, Bruce Southey wrote: > On 12/22/2009 09:05 AM, David Cournapeau wrote: > >> Hi, >> >> I have just released the 2nd release candidate for numpy 1.4.0, which >> fixes a few critical bugs founds since the RC1. Tarballs and binary >> installers for numpy/scipy may be found on >> https://sourceforge.net/projects/numpy. >> >> cheers, >> >> David >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > Hi, > This still crashes Python 2.7 with the test_multiarray.TestIO.test_ascii. > > The file numpy/core/src/multiarray/numpyos.c needs a change as per this > thread: > "test_multiarray.TestIO.test_ascii segmentationfault with Python2.7" > http://mail.scipy.org/pipermail/numpy-discussion/2009-December/047481.html > > The segmentation fault is avoided with this patch derived from the current > numpy-1.4RC2 version and not the SVN version. > > The patch looks ok, but the functions handle errors differently and I wonder if that has been completely audited. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
On Tue, Dec 22, 2009 at 9:05 AM, David Cournapeau wrote: > Hi, > > I have just released the 2nd release candidate for numpy 1.4.0, which > fixes a few critical bugs founds since the RC1. Tarballs and binary > installers for numpy/scipy may be found on > https://sourceforge.net/projects/numpy. > > cheers, > > David > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > This release results with the same import error on my system that I posted on http://old.nabble.com/Another-numpy-svn-installation-error-td26878029.html [gse...@ccn Desktop]$ python Python 2.6 (r26:66714, Jun 8 2009, 16:07:26) [GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "", line 1, in File "/home/gsever/Desktop/python-repo/numpy/numpy/__init__.py", line 132, in import add_newdocs File "/home/gsever/Desktop/python-repo/numpy/numpy/add_newdocs.py", line 9, in from lib import add_newdoc File "/home/gsever/Desktop/python-repo/numpy/numpy/lib/__init__.py", line 4, in from type_check import * File "/home/gsever/Desktop/python-repo/numpy/numpy/lib/type_check.py", line 8, in import numpy.core.numeric as _nx File "/home/gsever/Desktop/python-repo/numpy/numpy/core/__init__.py", line 6, in import umath ImportError: /home/gsever/Desktop/python-repo/numpy/numpy/core/umath.so: undefined symbol: npy_spacing Is there any remedy for this error? -- Gökhan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal for matrix_rank function in numpy
Hi, > I'm happy to write the doctests as tests. My feeling is there is no > objection to this function at the moment, so it would be reasonable, > unless I hear otherwise, to commit to SVN. Committed - with tests in tests_linalg.py - in revision 8029 Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] numpy 1.4.0 rc2
On 12/22/2009 09:05 AM, David Cournapeau wrote:
Hi,
I have just released the 2nd release candidate for numpy 1.4.0, which
fixes a few critical bugs founds since the RC1. Tarballs and binary
installers for numpy/scipy may be found on
https://sourceforge.net/projects/numpy.
cheers,
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
This still crashes Python 2.7 with the test_multiarray.TestIO.test_ascii.
The file numpy/core/src/multiarray/numpyos.c needs a change as per this
thread:
"test_multiarray.TestIO.test_ascii segmentationfault with Python2.7"
http://mail.scipy.org/pipermail/numpy-discussion/2009-December/047481.html
The segmentation fault is avoided with this patch derived from the
current numpy-1.4RC2 version and not the SVN version.
Bruce
--- numpyos.c 2009-12-22 09:36:58.0 -0600
+++ ../numpy-1.4.0rc2/numpy/core/src/multiarray/numpyos.c 2009-12-22
09:44:04.0 -0600
@@ -506,7 +506,11 @@
}
memcpy(buffer, s, n);
buffer[n] = '\0';
+#if PY_VERSION_HEX >= 0x0207
+result = PyOS_string_to_double(buffer, &q, NULL);
+#else
result = PyOS_ascii_strtod(buffer, &q);
+#endif
if (endptr != NULL) {
*endptr = (char*)(s + (q - buffer));
}
@@ -515,7 +519,11 @@
}
/* End of ##2 */
+#if PY_VERSION_HEX >= 0x0207
+return PyOS_string_to_double(s, endptr, NULL);
+#else
return PyOS_ascii_strtod(s, endptr);
+#endif
}
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] [ANN] numpy 1.4.0 rc2
Hi, I have just released the 2nd release candidate for numpy 1.4.0, which fixes a few critical bugs founds since the RC1. Tarballs and binary installers for numpy/scipy may be found on https://sourceforge.net/projects/numpy. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: PyTables 2.2b2 released
=== Announcing PyTables 2.2b2 === PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. This is the second beta version of 2.2 release. The main addition is the support for links. All HDF5 kind of links are supported: hard, soft and external. Hard and soft links are similar to hard and symbolic links in regular UNIX filesystems, while external links are more like mounting external filesystems (in this case, HDF5 files) on top of existing ones. This allows for a considerable degree of flexibility when defining your object tree. See the new tutorial at: http://www.pytables.org/docs/manual-2.2b2/ch03.html#LinksTutorial Also, some other new features (like complete control of HDF5 chunk cache parameters and native compound types in attributes), bug fixes and a couple of (small) API changes happened. In case you want to know more in detail what has changed in this version, have a look at: http://www.pytables.org/moin/ReleaseNotes/Release_2.2b2 You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://www.pytables.org/download/preliminary For an on-line version of the manual, visit: http://www.pytables.org/docs/manual-2.2b2 Resources = About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments === Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. **Enjoy data!** -- The PyTables Team -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] dot function or dot notation, matrices, arrays?
Christopher Barker wrote: > Dag Sverre Seljebotn wrote: >> This is readily done -- there is no computational portion except for >> what is in NumPy/Scipy or scikits, and I intend for it to remain that >> way. It's just another interface, really. >> >> (What kind of computations were you thinking about?) > > Nothing in particular -- just computational as opposed to symbolic > manipulation. OK. As a digression, I think it is easy to get the wrong impression of Sage that it is for "symbolics" vs. "computations". The reality is that the symbolics has been one of the *weaker* aspects of Sage (though steadily improving) -- the strong aspect is computations, but with elements that NumPy doesn't handle efficiently: Arbitrary size integer and rationals, polynomials (or vectors of their coefficients if you wish -- just numbers, not symbols), and so on. So the Sage design is very much about computation, it is just that the standard floating point hasn't got all that much attention. -- Dag Sverre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
