Re: [Numpy-discussion] Efficient ?axpy operation without copy (B += a*A)
On 2009-06-23, Michael McNeil Forbes wrote: > Is there a way of performing vectorized ?axpy (daxpy) operations > without making copies or dropping into C? > > i.e: I want to do > > big = (1,5000) > A = np.ones(big,dtype=float) > B = np.ones(big,dtype=float) > a = 1.5 > B += a*A I think the only available choice is to use the BLAS routines from scipy.lib: >>> from scipy.lib.blas import get_blas_funcs >>> axpy, = get_blas_funcs(['axpy'], [A, B]) >>> res = axpy(A.ravel(), B.ravel(), A.size, a) >>> res.base is B True >>> B[0,0] 2.5 Works provided A and B are initially in C-order so that ravel() doesn't create copies. If unsure, it's best to make use of the return value of axpy and not assume B is modified in-place. [clip] > There are exposed blas daxpy operations in scipy, but in the version I > have (EPD), these also seem to make copies (though recent version seem > to be fixed by looking at the source.) I don't see many relevant changes in scipy.lib recently, so I'm not sure what change you mean by the above. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] latex equations in docstrings
On 2009-06-24, Robert Kern wrote: [clip] > Yes. The HOWTO_BUILD_DOCS.txt is unfortunately out of date. So it is. Rewritten. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] fromfile and ticket #1152
On 2009-06-27, Jed Ludlow wrote:
> Of course, this discussion isn't new, and I don't know that it needs to be
> completely rehashed. See
>
> http://mail.scipy.org/pipermail/numpy-discussion/2009-May/042668.html
Especially this:
http://mail.scipy.org/pipermail/numpy-discussion/2009-May/042762.html
I think we sort of converged on some aspects of viable interface
options:
- "Non-strict" interface
Return as many items as can be read, or up to `count`, if given.
Never raise errors or warnings. Stop reading immediately on
unexpected input.
fromstring("1,2,x,4", sep=",") -> [1,2]
fromstring("1,2,x,4", sep=",", count=5) -> [1,2]
- Strict interface
Raise ValueError on malformed input, or if there are not
enough items for `count`.
fromstring("1,2,x,4", sep=",") -> ValueError
fromstring("1,2,3,4", sep=",", count=5) -> ValueError
The main disagreement was which of the above to use as the
default.
A hybrid of the above, which would raise an error only when
`count` was specified, was also suggested.
Then some variations on whether default values should be
introduced and what to do with non-numeric entries in this case.
I believe:
- We should not break backward compatibility, so the "non-strict" interface
should be the default.
No errors or warnings raised, except passing through underlying
I/O errors (eg. sector not found and other OS-level stuff).
- We could optionally, later on, implement the strict interface.
- We should drop the idea of default values for now, and
keep fromfile and fromstring simple.
[clip]
> I would vote toward harmonizing the behavior with the python built-in
> fid.read(bytes) as the default, which simply returns as many items as could be
> read before the EOF was reached, including zero. An additional strict
> interface
> could be added (by keyword) for those who want an exception raised whenever
> the
> requested read count could not be completed.
+1
--
Pauli Virtanen
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] stderr
On 2009-06-27, Charles R Harris wrote:
[clip]
>
>PyOS_snprintf(format, sizeof(format), _FMT1, prec);
>res = numpyos_ascii_for...@type@(buf, buflen, format, val, 0);
>if (res == NULL) {
>fprintf(stderr, "Error while formatting\n");
>return;
>}
>
[clip]
> Do we want to raise an error here? Alternatively, we could use an assert.
I'd advise against asserts. They should be used only for
conditions that are (expected to be) logically impossible. This
one here seems to be possible when out-of-memory, or some other
condition.
Also, an assert makes the program crash on C-level, which is
clearly undesirable in a Python program as it cannot be handled.
Raising an error here seems to be the proper thing to do.
--
Pauli Virtanen
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.memmap and memory usage
Wed, 01 Jul 2009 10:17:51 +0200, Emmanuelle Gouillart kirjoitti: > I'm using numpy.memmap to open big 3-D arrays of Xray tomography > data. After I have created a new array using memmap, I modify the > contrast of every Z-slice (along the first dimension) inside a for loop, > for a better visualization of the data. Although I call memmap.flush > after each modification of a Z-slice, the memory used by Ipython keeps > increasing at every new iteration. At the end of the loop, the memory > used by Ipython is of the order of magnitude of the size of the data > file (1.8Go !). I would have expected that the maximum amount of memory > used would corresponde to only one Z-slice of the 3D array. See the code > snapshots below for more details. > > Is this an expected behaviour? How can I reduce the amount of > memory used by Ipython and still process my data? How do you measure the memory used? Note that on Linux, "top" includes the size of OS caches for the memmap in the RSS size of a process. You can try to monitor "free" instead: $ free total used free sharedbuffers cached Mem: 12300488 11485664 814824 0 6429287960736 -/+ buffers/cache:28820009418488 Swap: 7847712 24287845284 If the memory is used by OS caches, the "used" number on the "-/+ buffers/ cache" line should stay constant while the program runs. In this case, what is most likely actually taking up memory is the OS buffering the data in memory, before writing it to disk. Linux has at least some system-wide parameters available that tune the aggressiveness of data cachine. I suppose there may also be some file-specific settings, but I have no idea what they are. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] make html
On 2009-07-01, Nils Wagner wrote: > I am using > >>>> numpy.__version__ > '1.4.0.dev7094' > > make html yields > > /home/nwagner/svn/numpy/doc/source/reference/generated/numpy.trunc.rst:: > WARNING: document isn't included in any toctree > done > preparing documents... done > Exception occurred: 2%] reference/generalized_ufuncs >ures >File > "/home/nwagner/local/lib64/python2.6/site-packages/Sphinx-0.6.1-py2.6.egg/sphinx/environment.py", > > line 921, in get_toc_for > toc = self.tocs[docname].deepcopy() > KeyError: 'reference/generalized_ufuncs' Do "make clean" first. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy complex types, packing and C99
On 2009-07-01, Charles R Harris wrote: > On Wed, Jul 1, 2009 at 1:59 AM, David Cournapeau > wrote: >>I would like to add an explicit configuration test to check that our >> complex type is compatible with the C99 complex type (when available). >> Is this ok? Seems OK to me. >>As currently defined, complex c types (npy_cfloat, etc...) are not >> defined in a way such as they are binary compatible with the C99 complex >> type. Strictly speaking, packing the complex type is an ABI break, but >> we already make the assumption in ufunc, so we would have wrong >> results/crashes currently if it were not packed, so I believe the check Is there a reason not to pack our complex number struct? I think if we bump the ABI version, changing this should be OK. >> is harmless by itself. The rationale is that I would like to add support >> for complex math in npy_math (cabs, cexp, etc...). As I would like to >> use the C99 complex type (when available), this requires that numpy >> complex type is ABI compatible with C99 type. Useful goal: this would allow using npy_math functions as a compatibility fallback when the C99 ones are not available. > What do you mean by "packing" ? C99 defines a complex type such that the real and imag parts are packed as in an array: double complex t; real = ((double*)&t)[0]; imag = ((double*)&t)[1]; Numpy defines this type as a struct, which may imply additional padding. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy complex types, packing and C99
On 2009-07-01, Charles R Harris wrote:
[clip]
> OK. I don't see a problem then. As David says the ufuncs
> already work that way. Hmm... do we want to
> implement creal{l,,f} and cimag{l,,f} also?
They might come useful.
> They are built in functions in gcc, can we detect that?
Don't know, probably we can?
> Anyway, those functions would make rewriting the current
> complex functions pretty easy. Or do we want to rewrite the
> current functions to be ABI compatible with c99?
I think that for scipy.special this would be useful. Of course,
the operator semantics for complex numbers can't be used there,
but still sticking closer to C99 could be useful.
A separate question is whether we want to implement Numpy's
complex ufuncs using the C99 ones when they are available.
I had a branch for this:
http://github.com/pv/numpy-work/tree/c99-complex-umath
This has some risks: the system-provided complex-valued functions
may be broken in different ways, or suffer from some subtle
flaws. This is likely more common than having broken real-valued
functions that have been around quite a while longer. (To give an
example: casinh(1e-20) == 0 with GNU Libc 2.9.)
--
Pauli Virtanen
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy complex types, packing and C99
On 2009-07-02, David Cournapeau wrote: > I think I will merge the complex_umath_tests branch soon > (platform-specific failures on build bot will be interesting), unless > someone sees a problem with it. I think we tried this already (my c99-umath-funcs branch had TestC99 special case tests that were in Numpy trunk for a while). The outcome was that the current implementations of the complex functions don't have essentially any special-case behavior that's consistent across different platforms. And our buildbot coverage is not large enough, so users will report failing tests even if buildbots are OK... After repeating this cycle a couple of times, IIRC only some special cases of log survived :) Of course, if you meant to merge the tests first to the new implementations and that to trunk, this sounds better. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] roots and high-order polynomial
On 2009-07-03, Charles R Harris wrote:
> roots? The connection between polynomial coefficients and polynomial values
> becomes somewhat vague when the polynomial degree becomes large, it is
> numerically ill conditioned.
In addition to switching to higher precision than machine
precision, another alternative could be choosing a stabler (eg.
Chebyshev) polynomial basis than the {1,x,x^2,...} one.
For the Chebyshev series f(x) = sum_{j=0}^N a_j T_j(x), the
companion matrix is (check!)
A = zeros((N, N))
A[0,1] = 1
j = arange(1, N-1)
A[j, j+1] = 0.5
A[j, j-1] = 0.5
A[-1,:] = -0.5*a[:-1]/a[-1]
A[-1,-2] += 0.5
and the zeros are
x = linalg.eig(A)[0]
See eg. http://dx.doi.org/10.1016/j.apnum.2005.09.007 for more.
--
Pauli Virtanen
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Adding new functions to Numpy
Hi, When you add new functions to Numpy, please include .. versionadded:: 1.4.0 in the Notes section of the function's docstring, and add the functions to an appropriate section of the reference guide. Like this: http://projects.scipy.org/numpy/changeset/7107 This way, users can keep track of which features are supported in which version. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with ML subscriptions?
2009-07-04 22:52 +0200, Fons Adriaensen wrote: [clip] > I subscribed to numpy-discussion almost two days ago. > I do receive messages from the list, but the ones I post > don't appear. I wrote to numpy-discussion-ow...@scipy.org > but got no response so far. So it looks that either this > list doesn't want me or there is some other problem. > Anyway it seems impossible to reach the right people. I don't think I am able to help you here. Forwarding this to the ML, so the list admins know. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] The baffling behavior that just won't die
Sun, 05 Jul 2009 20:27:02 -0700, d_l_goldsmith kirjoitti: [clip] > c:\Python25\Lib\site-packages\numpy>python Just don't run Python inside Numpy's package directory. This is not Numpy- specific: doing a thing like that just breaks relative imports. Also, Robert's answer in the old thread applies also here: instead of a file random.py, you have a package named 'random' in the working directory. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Mathematica vs Matlab vs Python
Tue, 07 Jul 2009 21:30:06 -0500, alan kirjoitti: > Mathematica vs Matlab vs Python > > http://www.larssono.com/musings/matmatpy/index.html The Python code there is not very idiomatic Numpy code. It's written for Numeric, and fancy indexing etc. are not used. Seems like the author also left it as an exercise for the reader to implement the data file parser in Matlab. I doubt it can be easily done in 20 lines :) -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)
Hi, Ticket #1143 points out that Numpy's reduction operations are not always cache friendly. I worked a bit on tuning them. Just to tickle some interest, a "pathological" case before optimization: In [1]: import numpy as np In [2]: x = np.zeros((8, 256)) In [3]: %timeit x.sum(axis=0) 10 loops, best of 3: 850 ms per loop After optimization: In [1]: import numpy as np In [2]: x = np.zeros((8, 256)) In [3]: %timeit x.sum(axis=0) 10 loops, best of 3: 78.5 ms per loop For comparison, a reduction operation on a contiguous array of the same size: In [4]: x = np.zeros((256, 8)) In [5]: %timeit x.sum(axis=1) 10 loops, best of 3: 88.9 ms per loop Funnily enough, it's actually slower than the reduction over the axis with the larger stride. The improvement factor depends on the CPU and its cache size. The code is here: http://github.com/pv/numpy-work/ticket/1143-speedup-reduce A benchmark script numpy/core/src/bench-reduce.py is included. Just do ./bench-reduce.py ../../../build/lib.linux-i686-2.6 | tee output.txt to run it against the system Numpy, and ./bench-reduce.py plot < output.txt to plot the results. Here's one result set: http://www.iki.fi/pav/tmp/xeon.png http://www.iki.fi/pav/tmp/xeon.txt The y-axis is proportional to the number of elemental (addition) operations per second, x-axis the total size of the array. Vertical lines show how the result changed by the cache optimization. Factor of 2 improvement seems common in the test cases (typically, arrays that are "skinny" in some dimension). There are also some factors of 10, especially for arrays larger than the 6144 kb cache of this particular CPU. Also, for this CPU, there are no significant regressions. On an older CPU (slower, smaller cache), the situation is slightly different: http://www.iki.fi/pav/tmp/athlon.png http://www.iki.fi/pav/tmp/athlon.txt On average, it's still an improvement in many cases. However, now there are more regressions. The significant ones (factor of 1/2) are N-D arrays where the reduction runs over an axis with a small number of elements. The optimization is in evaluating the reduction operation with a blocked loop (pseudocode) for block_idx in loop_over_blocks_except_axis(arr): result[block_idx] = arr[block_idx,0] for j in xrange(1, N): result[block_idx] += arr[block_idx,j] } ufunc loop instead of the usual ufunc reduction loop for idx in loop_over_array_except_axis(arr): result[idx] = arr[idx,0] } for j in xrange(1, N): } ufunc loop result[idx] += arr[idx,j] } The blocks are chosen to minimize the stride of the inmost ufunc loop, maximizing reuse of elements fitting in a single cache line. This has to be balanced against avoiding inner loops over very small dimensions, which incur some overhead. For N-D arrays, I also attempt to lump several dimensions into the ufunc loop, if the data can be iterated over with a single stride. The decision heuristics are in ufunc_object.c:_construct_reduce, and the new outer loop NOBUFFER_UFUNCREDUCE_TRANSPOSE in ufunc_object.c:PyUFunc_Reduce. Unfortunately, improving the performance using the above scheme comes at the cost of some slightly murky heuristics. I didn't manage to come up with an optimal decision rule, so they are partly empirical. There is one parameter tuning the cross-over between minimizing stride and avoiding small dimensions. (This is more or less straightforward.) Another empirical decision is required in choosing whether to use the usual reduction loop, which is better in some cases, or the blocked loop. How to make this latter choice is not so clear to me. The present approach seems to mostly work, as far as the benchmark produces improvements and produces no severe regressions. However, I think the regressions for the older CPU point out that something still needs to be tuned. The main suspect is the latter decision rule, but I'm not sure what the correct solution here is. Comments? I'm not a cache optimization expert, so bright ideas about better decision rules and further optimizations would be appreciated. Also, I think this addition could be nice to have included in Numpy. The main value IMHO is in boosting the most cache-inefficient corner cases. I think many users do not realize that np.zeros((256, 8)).sum(axis=1) is currently more efficient than np.zeros((8, 256)).sum(axis=0). And as shown above, there is no real reason to have this performance difference... Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)
On 2009-07-08, Charles R Harris wrote:
[clip]
> How do the benchmarks compare with just making contiguous copies? Which is
> blocking of sort, I suppose.
I think that's slower than just walking over the discontiguous
array:
1) The new code: (on the Athlon machine)
$ ./bench-reduce.py ../../../build/lib.linux-i686-2.6 8000,260 0,1 0
<8000> {260}[ <2080> {8} ]
-15 vs. 0.876923 : 1 2
new 8000,260 0,1 0 27.7 29.5 28.9 30.2 30.3
2) Old code, with "x.transpose(1,0).copy().sum(axis=-1)":
$ ./bench-reduce.py '' -o 8000,260 0,1 0
old 8000,260 0,1 0 5.03 5.13 5.11 5.07 4.87
3) Old code, with "x.sum(axis=0)"
$ ./bench-reduce.py '' 8000,260 0,1 0
old 8000,260 0,1 0 18.6 17.3 16.6 18.9 17.2
Last five numbers on the rows are repeats per second, five
repeats. (Above, the trailing dimension 260 was chosen so that it
maximizes the speed of the old code: 260 is almost twice faster
than 256... This seems to be a cache padding effect.)
I think this is expected: the main bottleneck is probably the
memory bandwidth, so when you make a copy and then reduce it, you
round-trip the array twice through the CPU cache. This is likely
to be more expensive than a single trip, even if it's
discontiguous.
Also, I'm not sure if copying a discontiguous array in Numpy
actually has any cache optimization, or whether it just walks the
array in virtual C-order copying the data. If so, then it
probably hits similar cache problems as the non-optimized
reduction operation.
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)
On 2009-07-08, Stéfan van der Walt wrote: > I know very little about cache optimality, so excuse the triviality of > this question: Is it possible to design this loop optimally (taking > into account certain build-time measurable parameters), or is it the > kind of thing that can only be discovered by tuning at compile-time? > ATNumPy... scary :-) I'm still kind of hoping that it's possible to make some minimal assumptions about CPU caches in general, and have a rule that decides a code path that is good enough, if not optimal. I don't think we want to go the ATNumPy route, or even have tunable parameters chosen at build or compile time. (Unless, of course, we want to bring a monster into the world -- think about cross-breeding distutils with the ATLAS build system :) -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performing operations in-place in numpy
On 2009-07-08, Charles R Harris wrote: > In that case I don't see a problem offhand. That said, I haven't looked at > the code yet. I'm a bit worried about the problem that cropped up in the ticket with the complex ufuncs. As Luca noted in the ticket, obj3 = PyArray_Conjugate((PyAO *)obj1, NULL); needed to be replaced with Py_INCREF(obj1); obj3 = PyArray_Conjugate((PyAO *)obj1, NULL); Py_DECREF(obj1); to avoid overwriting obj1 when refcount of `obj1 == 1`. Now, having refcounts of 1 can be a common condition in arrays in C code, and PyArray_Conjugate is a public function in Numpy's C-API. So, as it stands now, this patch seems to potentially break existing C-extension modules that use the Numpy/Numeric API. This would need to be fixed. Perhaps the INCREF/DECREF pair can be moved inside PyArray_Conjugate. If I understand correctly similar issues can only crop up in functions calling the *GenericUnary* etc. ufunc functions directly. Are these functions part of the public API? If not, there's a chance of fixing their callers inside Numpy. If they are a part of the public API, then the issue seems to become more hairy. Anyway, it seems there is some auditing to do before this change can be safely considered. Also, the speedups obtained were fairly modest, 20%. Are they larger for more complicated expressions? (Ie. is there an expression whose execution time is halved?) -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performing operations in-place in numpy
Thu, 09 Jul 2009 10:00:25 +0200, Matthieu Brucher kirjoitti: > 2009/7/9 Citi, Luca : >> Hello >> >> The problem is not PyArray_Conjugate itself. The problem is that >> whenever you call a function from the C side and one of the inputs has >> ref_count 1, it can be overwritten. This is not a problem from the >> python side because if the ufunc sees a ref_count=1 it means that no >> python object is referencing to it. > > Does this also hold if you are using the Numpy API directly? Say I've > decided to write some operation with the Numpy API, I will never have > one of my arrays with ref_count == 1? Newly created arrays have a refcount of 1, so this is a problem for the C-API. What needs to be ensured in Numpy is that none of the C-API functions can cause unexpected modification of their input arguments. So one would have to go through the list of exported functions, and check that none of them is problematic. (We know already that at least PyArray_Conjugate is.) -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)
Thu, 09 Jul 2009 09:54:26 +0200, Matthieu Brucher kirjoitti: > 2009/7/9 Pauli Virtanen : [clip] >> I'm still kind of hoping that it's possible to make some minimal >> assumptions about CPU caches in general, and have a rule that decides a >> code path that is good enough, if not optimal. > > Unfortunately, this is not possible. We've been playing with blocking > loops for a long time in finite difference schemes, and it is always > compiler dependent (that is, the optimal size of the block is bandwidth > dependent and even operation dependent). I'm not completely sure about this: the data access pattern in a reduce operation is in principle relatively simple, and the main focus would be in improving worst cases rather than being completely optimal. This could perhaps be achieved with a generic rule that tries to maximize data locality. Of course, I may be wrong here... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performing operations in-place in numpy
Thu, 09 Jul 2009 10:03:47 +0100, Citi, Luca kirjoitti: [clip] > Excuse me if I insist, PyArray_Conjugate is not the problem. If when > using the numpy API, it is accepted something like: > > obj1 = PyArray_CreateSomehowAnArray(); > obj2 = PyArray_DoSomethingWithArray(obj1,...); > obj3 = PyArray_DoSomethingElseWithArray(obj1,...); > Py_DECREF(obj1); > > then there is no way my patch is guaranteed to not break things. So it seems. PyUFunc_GenericFunction is a part of the public C-API, so I guess this discussion is moot. I doubt we can change ufunc semantics at this point, and I don't see ways in which one could distinguish between "temporary arrays" and refcount-1 arrays used in extension modules. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performing operations in-place in numpy
On 2009-07-09, Citi, Luca wrote: > Let me see if I understand correctly... > what you suggest is something like: > 1) adding an argument flag to construct_arrays > that enables/disables the feature > 2) adding the same argument flag to construct_loop which > is passed untouched to construct_arrays > 3) set the flag to "disable" in the construct_loop call inside > PyUFunc_GenericFunction > 4) write an exact copy of PyUFunc_GenericFunction with the flag > "enabled" > 5) in ufunc_generic_call, call the latter instead of > PyUFunc_GenericFunction > Am I correct? > > Sounds doable to me as long as ufunc.__call__ is not > directly or indirectly [used] in the numpy C API. It's conceivable that someone would call PyObject_Call(ufunc, args, kw) from C code. But this is already a bit contrived, given that PyUFunc_GenericFunction is there, and the args tuple probably usually has new references of the contained objects. [CPython itself creates a new tuple when doing a = (np.zeros((4,)), np.zeros((5,))) np.add(*a) so that also in this case the refcounts of the two arrays in the tuple are > 1.] -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] speed up np.diag
Fri, 10 Jul 2009 15:55:58 +0100, Citi, Luca kirjoitti: [clip] > ## SUGGESTED > def diag(v, k=0): > v = asarray(v) > s = v.shape > if len(s) == 1: [clip] > elif len(s) == 2: > if v.flags.f_contiguous: > v, k, s = v.T, -k, s[::-1] Is this correct? The .flat iterator always traverses the array in virtual C-order, not in the order it's laid out in memory. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Optimizing reduction loops (sum(), prod(), et al.)
Wed, 08 Jul 2009 22:16:22 +, Pauli Virtanen kirjoitti: [clip] > On an older CPU (slower, smaller cache), the situation is slightly > different: > > http://www.iki.fi/pav/tmp/athlon.png > http://www.iki.fi/pav/tmp/athlon.txt > > On average, it's still an improvement in many cases. However, now there > are more regressions. The significant ones (factor of 1/2) are N-D > arrays where the reduction runs over an axis with a small number of > elements. Part of this seemed (thanks, Valgrind!) to be because of L2 cache misses, which came from forgetting to evaluate also the first reduction iteration in blocks. Fixed -- the regressions are now less severe (most are ~0.8), although for this machine there are still some... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.isfinite on structured arrays
Tue, 14 Jul 2009 14:45:11 -0400, Pierre GM kirjoitti:
> Consider the following code:
> >>> a = np.array(zip(np.arange(3)),dtype=[('a',float)]) np.isfinite(a)
> NotImplemented
>
> That is, when the input is a structured array, np.isfinite returns an
> object of type NotImplementedType. I would have expected it to raise a
> NotImplementedError exception.
>
> Can anybody cast some light on this issue ? Thanks a lot in advance
Seems like a bug. As I understand, NotImplemented is intended to be
returned only from __lt__ etc. comparison methods.
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] saving complex vector
Wed, 15 Jul 2009 10:05:12 -0400, Neal Becker kirjoitti:
> Simple question. I want to save a complex vector as text in format
>
> real_0 imag_0\n
> real_1 imag_1\n
> ...
>
> I thought to use something like:
> np.savetxt ('gen_qpsk.txt', (mod_out.real, mod_out.imag), fmt='%g %g\n')
>
> I need a way to reinterpret the complex data as an array with 2 columns
> to make this work. Ideas?
Try np.c_[mod_out.real, mod_out.imag]
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Record arrays, nesting, and assignment
On 2009-07-15, Robert Kern wrote:
> On Wed, Jul 15, 2009 at 14:19, Vebjorn Ljosa wrote:
>> Suppose I have a record array where one of the fields is a nested array:
>>
>> >>> from numpy import *
>> >>> desc = dtype([('point', 'i4', 3), ('unimportant', 'S3')])
>> >>> a = array([((1,2,3), 'foo'), ((7,8,9), 'bar')], dtype=desc)
>> >>> a
>> array([([1, 2, 3], 'foo'), ([7, 8, 9], 'bar')],
>> dtype=[('point', '> >>> a[0]
>> ([1, 2, 3], 'foo')
>>
>> If I try to assign to a[0]['point'], it only works partially:
>>
>> >>> a[0]['point'] = array([4, 5, 6])
>> >>> a[0]
>> ([4, 2, 3], 'foo')
>
[clip]
>
> Generally, scalars are never views. a[0] pulls out a record scalar.
> Assigning into that scalar does not affect the original memory.
> a['point'] creates an array that is a view and assigning to the [0]
> element of that array modifies the original memory.
But then, why does it alter the first element of the sub-array?
This seems like a bug...
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding new functions to Numpy
On 2009-07-16, David Cournapeau wrote:
> Hi Pauli,
>
> On Sat, Jul 4, 2009 at 9:59 PM, Pauli Virtanen wrote:
>> Hi,
>>
>> When you add new functions to Numpy, please include
>>
>> .. versionadded:: 1.4.0
>
> What is the best way to do this in the reference guide directly as
> well (for C API). For example, I added the function npy_copysign for
> 1.4.0, documented in the c-api.coremath.rst, but I don't know where to
> put the versionadded.
In the Python docs, the versionadded:: typically is the last item
inside the function:: directive.
> More generally, I would like to follow our
> docstring format when documenting C functions, but how can I put the
> different sections in there (Parameters, Notes, etc...)
Yeah, currently there's no way to do that. I can probably add
numpy*function/cfunction/... etc. directives that allow this.
An alternative is to use
.. rubric:: Notes
to make the headings to appear.
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] VIGRA, NumPy and Fortran-order (again)
On 2009-07-17, Hans Meine wrote: [clip] > As discussing in-depth in [1], numpy does not support Fortran order very > well. > First, there are performance issues, but even more important: the order is > not > preserved when performing simple operations. :-( [clip] > The specs would be: preserve the *ordering* of the strides (i.e. we're using > mixed-order for RGB images to be able to write image[x, y] = (r, g, b)), and > in the case of multiple input arguments, use the same rules (i.e. array > priority) as for the output type determination. > > If I understood Travis' comments in the above-mentioned thread [1] correctly, > this would already fix some of the performance issues along the way (since it > would suddenly allow the use of special, optimized code paths). I was wondering about this too, when working on improving the cache coherency of the reduction operations. Also these would be more efficient if the striding of the output array could be chosen freely. I wonder if it would be OK to make this change... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding new functions to Numpy
On 2009-07-18, David Cournapeau wrote: > On Fri, Jul 17, 2009 at 4:11 AM, Pauli Virtanen wrote: [clip] >> Yeah, currently there's no way to do that. I can probably add >> numpy*function/cfunction/... etc. directives that allow this. >> >> An alternative is to use >> >> .. rubric:: Notes >> >> to make the headings to appear. > > Ok, that's better than nothing for the time being. I would really like > to have a plugin to have autodoc for C code as well as python code, > though. One more thing on the TODO list :) Well, you can now use ``np-cfunction::`` for this. I'm not 100% sure this is a wise route to go, but oh well... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] My identity
On 2009-07-20, Keith Goodman wrote: [clip] > Oh, sorry, I misunderstood. Yes, a similar change was made to eye but > not to identity. Nasty, duplicated code there it seems... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Functions to pack/unpack bytes?
Wed, 29 Jul 2009 10:20:19 -0400, Neal Becker kirjoitti: [clip] > Can 'view' switch byteorder? Doesn't seem to work: [clip] I think not: view only alters the dtype, ie., the interpretation of the block of memory. It does not physically reorder the data in memory, which is necessary for unpacking to a different byte order. The reliable way to unpack seems to be np.asarray(a, dtype='http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Internal Math Library of Numpy
On 2009-07-30, Nanime Puloski wrote: > Does Numpy use Python's standard math library when calculating > elementary functions such as exp(x) and acos(x)? No. > Also, what is the internal library of Numpy and Python's > standard math library? The standard C library, plus custom implementations for functions that are missing in it. > Are they platform independent? In practice, yes. There are some possible obscure corner cases currently in complex-valued inf/nan handling, though. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] itemsize() doesn't work
Fri, 21 Aug 2009 04:47:09 +, David Goldsmith wrote: [clip] > > > http://numpy.scipy.org/numpydoc/numpy-10.html [clip] > Is this editable through the Wiki? I went to the > Docstrings page and searched for "numpydoc" and "tutorial" and got no > hits. This is the old Numeric module documentation. It probably doesn't describe all points of Numpy accurately. Of course, the URL is misleading... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What type should / return in python 3k when applied to two integer types?
Fri, 28 Aug 2009 09:46:39 -0400, Neal Becker kirjoitti: > Robert Kern wrote: > >> On Thu, Aug 27, 2009 at 14:22, Christopher >> Barker wrote: >> >>> By the way -- is there something about py3k that changes all this? Or >>> is this just an opportunity to perhaps make some backward-incompatible >>> changes to numpy? >> >> Python 3 makes the promised change of int/int => float. > > Does that mean that we want numpy to do the same? I'm not so sure. > Sounds like opening a can of worms (numpy has more types to worry about > than just int and float. If we start playing strange games we may > regret it.) I believe we want to. This is not really a strange trick: it's just that in Python 3, the operator / is true_division, and // is floor_division. I believe any worms released by this are mostly small and tasty... The main issue is probably just choosing an appropriate float return type, and personally I believe this should be same as numpy's default float. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Question about np.savez
Tue, 01 Sep 2009 12:07:36 +, jorgesmbox-ml kirjoitti: > I know the documentation states that np.savez saves numpy arrays, so my > question relates to misusing it. Before reading the doc in detail, and > after reading about pickle and other options to make data persistent, I > passed np.savez a list of ndarrays. It didn't complain, but when I > loaded the data back, the list had been turned into an ndarray. Is this > behaviour expected? It did surprise me. Below there is an example: [clip] It is expected. savez casts its input to arrays before saving. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ticket for O(n) median function?
On 2009-09-01, Sturla Molden wrote: [clip] > I could not find any, so I'll ask if it's ok to create one. I have a > patch for /numpy/lib/function_base.py that uses any 'select' function to > obtain the median. I'll also submit the Cython code for quickselect. I'd say that just go ahead and create one -- there's little harm done even if it's a duplicate. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding a 2D with a 1D array...
Wed, 09 Sep 2009 13:08:22 +0200, Ruben Salvador wrote: > Perfect! Thank you very much :D > > It's not obvious, though...I think I should read more deeply into > Python/NumPy...but for the use I'm giving to it... > > Anyway, I thought the pythonic way would be faster, but after trying > with a size 8 instead of 8...the for loop is faster! > > Pythonic time ==> 0.36776400 seconds > For loop time ==> 0.31708717 seconds Doubtful: In [1]: import numpy as np In [2]: a = np.zeros((8, 26)) In [3]: b = np.zeros((8,)) In [4]: def foo(): ...: d = np.empty(a.shape, a.dtype) ...: for i in xrange(a.shape[0]): ...: d[i] = a[i] + b[i] ...: In [5]: %timeit d=a+b[:,None] 100 loops, best of 3: 18.3 ms per loop In [6]: %timeit foo() 10 loops, best of 3: 334 ms per loop ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Error in header file - wrong mailing list?
la, 2009-09-12 kello 23:57 +0200, Mads Ipsen kirjoitti: > Hey, > > I recently posted a bug related to a compile error in the header file > 'npy_common.h' but have received no responses so far. > > Am I posting this in the wrong mailing list? It is the correct list, but in general bug reports are better reported in the bug tracker: http://projects.scipy.org/numpy/ Message concerning bugs, especially if they are minor ones, are easily lost in the other traffic on the ML. The error you get from the comma at the end of the enum must be because you have -pedantic -Werror in your CFLAGS. Just unset CFLAGS or so before compilation. Yes, comma at the end of enum list is not strictly valid C89, although it is valid C99 and already passes for most compilers. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Error in header file - wrong mailing list?
su, 2009-09-13 kello 01:47 +0300, Pauli Virtanen kirjoitti: [clip] > The error you get from the comma at the end of the enum must be because > you have > > -pedantic -Werror > > in your CFLAGS. Just > > unset CFLAGS > > or so before compilation. Yes, comma at the end of enum list is not > strictly valid C89, although it is valid C99 and already passes for most > compilers. Another possibility is that these flags come from /usr/lib/pythonX.Y/config/Makefile -- in that case it's maybe possible to override one of the the BASECFLAGS etc. variables with environment vars. Also, I see that OPT="-std=c89 -pedantic -Werror" python setup.py won't succeed in current SVN, because apparently the configuration detection code is not strict C89. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Error in header file - wrong mailing list?
Mon, 14 Sep 2009 09:08:34 +0200, Mads Ipsen wrote: [clip] > Well, I don't know you consider this as a valid argument, but to me its > a matter of removing a single comma, which will make the source less > sensitive to compilers and compiler flags. That's done. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] defmatrix move - new docstrings disappeared
to, 2009-09-17 kello 11:21 -0400, Ralf Gommers kirjoitti: > After the recent move of the matrix module, all the changes to the > docstrings have disappeared from the doc wiki. I imagine the changes > still live somewhere in the underlying repo but are not visible to the > user. Can they be restored? If there is some unforeseen problem with > that, I do have a recent patch saved that contains the latest edits > (but of course it won't apply cleanly). They are not lost (the app is designed so that things are not easily lost), and can be found by writing the old full URL: http://docs.scipy.org/numpy/docs/numpy.core.defmatrix/ The easiest way to make the transition is to copy the docstrings via SQL on the server. I just copied the numpy.core.defmatrix.* docstring revisions and comments to numpy.matrx.defmatrix.*. For reference, it can be done with two SQL statements on the database: insert or ignore into docweb_docstringrevision (docstring_id, text, author, comment, timestamp, review, ok_to_apply) select 'numpy.matrx.defmatrix' || substr(docstring_id, 21, length(docstring_id)), text, author, comment, timestamp, review, ok_to_apply from docweb_docstringrevision where docstring_id like 'numpy.core.defmatrix%'; insert or ignore into docweb_reviewcomment (docstring_id, text, author, timestamp, resolved) select 'numpy.matrx.defmatrix' || substr(docstring_id, 21, length(docstring_id)), text, author, timestamp, resolved from docweb_reviewcomment where docstring_id like 'numpy.core.defmatrix%'; Moving would have been two one-liners, but copying maybe better conforms to the append-only policy. Also, the sqlite version on the server doesn't support a replace() function, yuck. I think everything should be OK now -- please let me know if something is broken. > I saved that patch from the wiki because I was worried about the large > number of unmerged changes, there are about 500 docstrings with > changes right now (close to 50% of all important docstrings). Would it > make sense to make sure (maybe with an automatic reminder email) that > changes from the wiki get merged if there are enough changes - 100 > changed docstrings or changes older than two months, say? I'll put the following to my crontab, let's see if it helps... 15 14 * * sun test `wget -q -O- 'http://docs.scipy.org/numpy/patch/' | grep -c ''` -lt 100 || echo "Hey man, lots of stuff is waiting at http://docs.scipy.org/numpy/patch/"; | mail -s "Lots o' Stuff waiting in Pydocweb" pa...@localhost Anyone else with SVN commit access is welcome to do similarly :) > I know it is quite a bit of work to do the merge. I remember Pauli > saying that most of the work was reviewing all changes to make sure > they are an improvement over the current svn version. Is that correct? > I can help with that if necessary. You have reviewer rights, so you can go and click the "OK to apply" buttons, to flag those changes that are better than SVN. Also, check that no unwanted or dangerous code (eg. careless handling of temporary files, etc) is added to the examples -- they may eventually be executed after committed to SVN. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] defmatrix move - new docstrings disappeared
to, 2009-09-17 kello 18:19 +0200, Scott Sinclair kirjoitti: [clip] > It's probably important that the documentation patches should be > committed pretty soon after being reviewed for obvious malicious code > and marked "OK to Apply". It's possible to edit docstrings that are > marked as "OK to apply", without this flag being removed. If that's possible, then it's a bug. But I don't see how that can happen -- do you have an example how to do this kind of edits that don't reset the ok_to_apply flag? > If none of the current commiters have time to commit doc patches on > demand, then perhaps it makes sense to give commit access to someone > working actively on the documentation (Ralf, DavidG ?). This could be > on the understanding that only doc patches would be commited by this > person. > > It's always going to be a lot of work when we let the trunk and > doc-editor get too far out of sync. The work scales linearly with the size of the diff to SVN, so it's not extremely bad. Of course, it's a bit of a work to go through hundreds of potential docstrings in one sitting. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] defmatrix move - new docstrings disappeared
Thu, 17 Sep 2009 21:16:12 -0400, Ralf Gommers wrote: > There is a lot more it turns out: > > atleast_1d > atleast_2d > atleast_3d > hstack > vstack > correlate2 > linspace > logspace > finfo > iinfo > MachAr So it seems -- David's been moving stuff around lately. > Based on this I suspect there is quite a bit of work that got lost > earlier in the summer. A couple of times I saw the count of "needs > editing" in the stats go up by several or even several tens. At the time > I thought those were all objects that were new to NumPy, but more likely > they got moved around and the doc writers redid the work. I doubt that -- not so much has been moved around. It's easy to see from "git log --stat -M -C" that only shape_base, getlimits, and machar have been moved around since 2007. The main cause for "Needs editing" increasing is that I elevated occasionally some items from "Unimportant" status that were actually important. If you have good ideas how to the "move/delete warnings" should appear in the web UI, and what the application should do to make it easy to fix them, go ahead and tell them. Designing the UI and how it should work is the first part of the work in making any improvements. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] defmatrix move - new docstrings disappeared
Fri, 18 Sep 2009 07:42:08 +, Pauli Virtanen wrote: [clip] > I doubt that -- not so much has been moved around. It's easy to see from > "git log --stat -M -C" that only shape_base, getlimits, and machar have > been moved around since 2007. A complete listing obtained from SQL: numpy-docs/reference/generalized_ufuncs.rst Taken care of when the move was made. numpy.core.defmatrix.asmatrix numpy.core.defmatrix.bmat numpy.core.defmatrix.matrix numpy.core.defmatrix.matrix.all numpy.core.defmatrix.matrix.any numpy.core.defmatrix.matrix.argmax numpy.core.defmatrix.matrix.argmin numpy.core.defmatrix.matrix.getA numpy.core.defmatrix.matrix.getA1 numpy.core.defmatrix.matrix.getH numpy.core.defmatrix.matrix.getI numpy.core.defmatrix.matrix.getT numpy.core.defmatrix.matrix.max numpy.core.defmatrix.matrix.mean numpy.core.defmatrix.matrix.min numpy.core.defmatrix.matrix.prod numpy.core.defmatrix.matrix.ptp numpy.core.defmatrix.matrix.std numpy.core.defmatrix.matrix.sum numpy.core.defmatrix.matrix.tolist numpy.core.defmatrix.matrix.var numpy.core.defmatrix.matrix_power Taken care of, these are here since I copied the old strings to numpy.matrixlib. numpy.core.numeric.acorrelate All features in acorrelate were merged to the main correlate function. No way to really avoid lost work here. numpy.float96 Platform issue -- it's just not available on all platforms, and so hits a corner case in the app. numpy.lib.arraysetops.intersect1d_nu numpy.lib.arraysetops.setmember1d numpy.lib.arraysetops.unique1d These were deprecated. Work lost, but no way to really avoid this. numpy.lib.function_base.unique numpy.lib.utils.issubdtype These were committed to SVN before move -- no work lost. numpy.ma.core.MaskedArray.torecords The function was merged to "toflex" in SVN. No work lost. numpy.ma.core.get_data numpy.ma.core.get_mask These were already taken care of when the move occurred. numpy.matlib.empty numpy.matlib.eye numpy.matlib.identity numpy.matlib.ones numpy.matlib.rand numpy.matlib.randn numpy.matlib.repmat numpy.matlib.zeros These are hidden by some misconfiguration. No work lost, I'll try to fix this when I find time for it. So, in summary, I'm happy to say that not much has been moved around during two years, and we have caught and fixed all the largest changes in a few days after when they have occurred. Big thanks to Ralf & Scott for noticing these! Reorganization of the codebases is sometimes necessary. The thing to improve here is adding a better move/delete tracking functionality to the web application. There's always a balance between doing things manually vs. writing an automated way -- here, an automated way might be better, as keeping track of and fixing these manually is somewhat fiddly work, even though it boils down to a few SQL statements. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] defmatrix move - new docstrings disappeared
Fri, 18 Sep 2009 11:24:25 -0400, Ralf Gommers wrote: [clip] >> If you have good ideas how to the "move/delete warnings" should appear >> in the web UI, and what the application should do to make it easy to >> fix them, go ahead and tell them. Designing the UI and how it should >> work is the first part of the work in making any improvements. >> >> > One possible solution: have a page with all unhandled moves/deletes, > similar to the Merge page. This page can list all relevant docstrings > and have buttons to copy over the old page, or to ignore the event. As > extra insurance, the newly created page could have a warning added to it > in a comment that there may be a problem. > > It may also be possible to copy over the whole page automatically but > that is more risky than clearly asking the user for a manual resolution. Sounds good, manual resolution for all new and removed pages is reasonably low-overhead, especially as there is no completely reliable way to detect when a new page is moved, or just new. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] I want to help with a numpy python 3.1.x port
pe, 2009-09-18 kello 23:41 -0600, Charles R Harris kirjoitti: [clip] > Hmm, that doesn't work for me, git finds no commits in common if I > pull and, as far as I can tell, essentially clones the whole thing. I > quess that's because I'm using git-svn and started in a different > place for the initial co and every commit to svn rebases. Diffs might > be the easiest way to go. Use our Git mirror if you need to collaborate: http://projects.scipy.org/numpy/wiki/GitMirror It's there precisely to allow a central tree around which to base your changes, and avoids precisely these problems. Also, avoid touching git-svn as much as possible. If you want to preserve interoperability, "git-svn dcommit" to SVN always from a separate commit branch, to which you cherry-pick changes from the branch you use to collaborate. This is because git-svn rewrites the commits you dcommit to SVN. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is ndarray.base the closest base or the ultimate base?
Mon, 21 Sep 2009 09:35:47 +0100, Citi, Luca wrote: > I cannot quite understand whether ndarray.base is the closest base, the > one from which the view was made or the ultimate base, the one actually > containing the data. > I think the documentation and the actual behaviour mismatch. The closest base. If the documentation says the opposite, it's wrong. That it's the closest base also causes crashes when the base chain becomes longer than the stack limit. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is ndarray.base the closest base or the ultimate base?
Mon, 21 Sep 2009 10:51:52 +0100, Citi, Luca wrote:
> Thanks for your quick answer.
>
> Is there a reason for that?
> Am I wrong or it only makes life harder, such as:
>
> while (PyArray_Check(base) && !PyArray_CHKFLAGS(base, NPY_OWNDATA)) {
>base = PyArray_BASE(base);
> }
>
> plus the possible error you underlined, plus the fact that this keeps a
> chain of zombies alive without reason.
>
> Are there cases where the current behaviour is useful?
I don't see real merits in the current behavior over doing the chain up-
walk on view creation. I don't know if anything in view creation requires
that the immediate base is alive afterwards, but that seems unlikely, so
I believe there's no reason not to make this change.
Some (bad) code might be broken if this was changed, for example
>>> y = x[::-1]
>>> y.base is x
but in practice this is probably negligible.
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy docstring sphinx pre-processors
ma, 2009-09-21 kello 13:35 -0400, Elaine Angelino kirjoitti: > ok a couple more questions: > 1) how does sphinxext relate to numpydoc? > sphinxext in scipy source tree -- > http://svn.scipy.org/svn/numpy/trunk/doc/sphinxext/ > numpydoc on PyPI -- http://pypi.python.org/pypi?% > 3Aaction=search&term=numpydoc&submit=search They are the same. If you want to use easy_install, use numpydoc. > > 2) what about postprocess.py, should i be using this too? > (http://svn.scipy.org/svn/numpy/trunk/doc/) It removes extra section headers from the Latex output. If you want to use it, you'll have to modify to match your module. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] something wrong with docs?
ma, 2009-09-21 kello 13:15 -0400, Skipper Seabold kirjoitti: > On Mon, Sep 21, 2009 at 7:27 AM, Neal Becker wrote: > > I'm trying to read about subclassing. When I view > > > > http://docs.scipy.org/doc/numpy/user/basics.subclassing.html?highlight=subclass#module- > > numpy.doc.subclassing > > > > It seems the examples show the _outputs_ of tests, but I don't see the > > actual example code. > > > > e.g., the first example renders like this: > > > > Simple example - adding an extra attribute to ndarray¶ > > Using the object looks like this: > > > > I'd like to see this sorted as well. The problem is that the > `testcode` directive > <http://docs.scipy.org/numpy/docs/numpy-docs/user/basics.subclassing.rst/> > is not recognized. I was recently a bit confused by this, and I went > to the rst file to view the code, but that's obviously not a fix for > the rendering problem. The `sphinx.ext.doctest` extension is not enabled, so the testcode:: etc. directives are not available. I'm not sure if it should be enabled -- it would be cleaner to just replace the testcode:: stuff with the ordinary example markup. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] something wrong with docs?
Mon, 21 Sep 2009 18:49:47 -0700, Fernando Perez wrote: > On Mon, Sep 21, 2009 at 11:32 AM, Pauli Virtanen wrote: >> The `sphinx.ext.doctest` extension is not enabled, so the testcode:: >> etc. directives are not available. I'm not sure if it should be enabled >> -- it would be cleaner to just replace the testcode:: stuff with the >> ordinary example markup. >> >> > Why not enable it? It would be nice if we could move gradually towards > docs whose examples (at least those marked as such) were always run via > sphinx. The more we do this, the higher the chances of non-zero overlap > between documentation and reality :) I think sphinx.ext.doctest is able to also test the ordinary >>> marked- up examples, so there'd be no large need for new directives. But oh well, I suppose enabling it can't hurt much. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] something wrong with docs?
Tue, 22 Sep 2009 23:15:56 -0700, David Goldsmith wrote: [clip] > "It would be nice if we could move gradually towards docs whose examples > (at least those marked as such) were always run via sphinx." Also the >>> examples are doctestable, via numpy.test(doctests=True), or enabling Sphinx's doctest extension and its support for those. What Fernando said about them being more clumsy to write and copy than separate code directives is of course true. I wonder if there's a technical fix that could be made in Sphinx, at least for HTML, to correct this... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Deserialized arrays with base mutate strings
Wed, 23 Sep 2009 09:15:44 +0200, Hrvoje Niksic wrote:
[clip]
> Numpy arrays with the "base" property are deserialized as arrays
> pointing to a storage contained within a Python string. This is a
> problem since such arrays are mutable and can mutate existing strings.
> Here is how to create one:
Please file a bug ticket in the Trac, thanks!
Here is a simpler way, although one more difficult to accidentally:
>>> a = numpy.frombuffer("A", dtype='S1')
>>> a.flags.writeable = True
>>> b = "A"
>>> a[0] = "B"
>>> b
'B'
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Deserialized arrays with base mutate strings
ke, 2009-09-23 kello 10:01 +0200, Hrvoje Niksic kirjoitti: [clip] > I guess this one could be prevented by verifying that the buffer is > writable when setting the "writable" flag. When deserializing arrays, I > don't see a reason for the "base" property to even exist - sharing of > the buffer between different views is unpreserved anyway, as reported in > my other thread. IIRC, it avoids one copy: ndarray.__reduce__ pickles the raw data as a string, and so ndarray.__setstate__ receives a Python string back. I don't remember if it's in the end possible to emit raw byte stream to a pickle somehow, not going through strings. If not, then a copy can't be avoided. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] chebyshev polynomials
to, 2009-09-24 kello 11:51 -0600, Charles R Harris kirjoitti: > Would it be appropriate to add a class similar to poly but instead > using chebyshev polynomials? That is, where we currently have [clip] Yes, I think. scipy.special.orthogonal would be the best place for this, I think. Numpy would probably be a wrong place for stuff like this. Ideally, all the orthogonal polynomial classes in Scipy should be rewritten to use more a stable representation of the polynomials. Currently, they break down at high orders, which is a bit ugly. I started working on something related in the spring. The branch is here: http://github.com/pv/scipy-work/tree/ticket/921-orthogonal but as you can see, it hasn't got far (eg. orthopoly1d.__call__ is effectively a placeholder). Anyway, the idea was to divide the orthopoly1d class to subclasses, each having more stable polynomial-specific evaluation routines. Stability-preserving arithmetic would be supported at least within the polynomial class. As a side note, should the cheby* versions of `polyval`, `polymul` etc. just be dropped to reduce namespace clutter? You can do the same things already within just class methods and arithmetic. Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] chebyshev polynomials
to, 2009-09-24 kello 14:31 -0500, Robert Kern kirjoitti: > On Thu, Sep 24, 2009 at 14:18, Pauli Virtanen wrote: > > As a side note, should the cheby* versions of `polyval`, `polymul` etc. > > just be dropped to reduce namespace clutter? You can do the same things > > already within just class methods and arithmetic. > > Just to clarify, you mean having classmethods that work on plain > arrays of Chebyshev coefficients? I'm +1 on that. I'm -1 on only > having a ChebyPoly class with instance methods, although it would be > useful to have as an adjunct to the plain routines. I meant only having a ChebyPoly class with instance methods. Personally, I've always used poly1d instances instead of the poly* routines apart from polyfit. But perhaps this is not how everyone uses them. Using class methods is an interesting idea, though. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] chebyshev polynomials
to, 2009-09-24 kello 13:53 -0600, Charles R Harris kirjoitti: [clip] > I was thinking of storing the chebyshev internally as the values at > the chebyschev points. This makes multiplication, differentiation and > such quite easy (resample and multiply/divide appropriatately). Its > equivalent to working in the fourier domain for convolution and > differentiation. The transform back and forth is likewise othogonal, > so stable. The intepolation also becomes simple using the barycentric > version. Sounds like you know this stuff well :) The internal representation of each orthogonal polynomial type can probably be whatever works best for each case. It should be no problem to sugar ChebyPoly up after the main work has been done. > As a side note, should the cheby* versions of `polyval`, > `polymul` etc. just be dropped to reduce namespace clutter? > You can do the same things already within just class methods > and arithmetic. > > What do you mean? The evaluation can use various stable methods > appropriate to the chebyshev series. This comment was just on the API -- the implementation of course should be appropriate. > I have a set of functions that does the first (works on > multidimensional arrays of coefficients, actually), but I am open to > ideas of what such a chebyschev class with these methods should look > like. An interval of definition should probably be part of the ctor. > Thoughts? Having the following features could be useful: - __call__, .roots, .order: as in poly1d - .data -> whatever is the internal representation - .coef -> Chebyshev coefficients? - .limits -> The interval - arithmetic: chebyshev chebyshev -> chebyshev - arithmetic: scalar chebyshev -> chebyshev - arithmetic: poly1d chebyshev -> chebyshev/poly1d (??) I'm not sure what __getitem__ and __array__ ought to return. Alternatives seem either to return the poly1d coefficients, or not to define these methods -- otherwise there will be confusion, especially if we want to use these in scipy.special.orthogonal. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Error building docs
Fri, 25 Sep 2009 12:29:30 -0400, Michael Droettboom wrote: > Anybody know why I might be seeing this? [clip] > Exception occurred:[ 0%] reference/arrays.classeslass > File "/home/mdroe/usr/lib/python2.5/site-packages/docutils/nodes.py", > line 471, in __getitem__ > return self.attributes[key] > KeyError: 'numbered' No ideas. I think I've seen KeyErrors before, but not from inside docutils. My only guess would be to remove the build directory and try again. If that does not help, I'd look into if there's some known issue with the current version of Sphinx's hg branch. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.numarray.transpose()
Mon, 28 Sep 2009 10:07:47 +0200, Michael.Walker wrote: [clip] > In [7]: f = f.transpose() > > In [8]: print f > [[1 3] > [2 4]] > > as expected. I mention this because I think that it is worth knowing > having lost a LOT of time to it. Is it worth filing as a bug report? Yes. It indeed seems that in numarray, transpose() transposes the array in-place. This could maybe be fixed by a new numarray-emulating ndarray subclass. The tricky problem then is that some functions don't, IIRC, preserve subclasses, which may lead to surprises. (Anyway, these should be fixed at some point...) At the least, we should write a well-visible "differences to numarray" document that explains all differences and known bugs. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.numarray.transpose()
Mon, 28 Sep 2009 09:29:30 -0500, Bruce Southey wrote: [clip] > This is not a bug! This specific difference between numpy and numarray > is documented on the 'converting from numarray' page: > http://www.scipy.org/Converting_from_numarray Oh. I completely missed that page. Now, it should just be transferred to the main documentation. Also, it might be possible to make numpy.numarray.ndarray different from numpy.ndarray. But I doubt this is high priority -- it may be more efficient just to document the fact. > What actually is incorrect is that the numpy.numarray.transpose has the > same docstring as numpy.transpose. So it would be very helpful to first > correct the numpy.array.transpose documentation. numpy.numarray.transpose is numpy.transpose, so fixing this would involve implementing the numarray-style transpose, too. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.numarray.transpose()
Tue, 29 Sep 2009 11:08:42 +0200, Michael.Walker wrote: [clip] > I am referring to the behaviour of numpy.numarray.transpose() being that > of numpy.transpose() instead of numarray.transpose. One expects that You probably mean the transpose methods numpy.numarray.ndarray.transpose and numarray.ndarray.transpose. The transpose functions function identically. > numpy.numarray would function as numarray, for the purpose of backwards > compatability. So, is this worth filing a bug report for? There is no harm in creating a bug ticket. This is at least a documentation issue even in the case the main problem becomes a wontfix. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] making the distinction between -0.0 and 0.0..
Tue, 29 Sep 2009 09:53:40 -0700, Christopher Barker wrote: [clip] > How can I identify -0.0? signbit -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] __array_wrap__
Tue, 29 Sep 2009 14:55:44 -0400, Neal Becker wrote: > This seems to work now, but I'm wondering if Charles is correct, that > inheritance isn't such a great idea here. > > The advantage of inheritance is I don't have to implement forwarding all > the functions, a pretty big advantage. (I wonder if there is some way to > do some of these as a generic 'mixin'?) The usual approach is to use __getattr__, to forward many routines with little extra work. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ufunc and errors
Wed, 30 Sep 2009 10:33:46 -0500, Robert Kern wrote: [clip] >> Also, will the arguments always be named x1, x2, x3, ..., or can I >> somehow give them custom names? > > The only place where names appear is in the docstring. Write whatever > text you like. The first line of the docstring is generated by Numpy and cannot be modified. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] merging docs from wiki
to, 2009-10-01 kello 12:19 -0400, Ralf Gommers kirjoitti: > Sorry to ask again, but it would really be very useful to get those > docstrings merged for both scipy and numpy. [clip] Numpy's new docstrings is are now in SVN too, for the most part. An amazing amount of work was done during the summer, thanks to all who participated! > > For numpy in principle the same procedure, except there are some > objects that need the add_newdocs treatment. There are two types of > errors, my question is (mainly to Pauli) if they both need the same > treatment or a different one. > > Errors: > 1. source location not known, like: > ERROR: numpy.broadcast.next: source location for docstring is not known > 2. source location known but failed to find a place to add docstrings, > like: > ERROR: Source location for numpy.lib.function_base.iterable known, > but failed to find a place for the docstring These I didn't commit yet. Mostly, they can be fixed by adding necessary entries to add_newdocs.py. However, some of these may be objects assigning docstrings to which may be technically difficult and requires larger changes. The second error may also indicate a bug in patch generation. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Easy way to test documentation?
ma, 2009-10-05 kello 13:54 -0600, Charles R Harris kirjoitti: > Is there an easy way to test build documentation for a module that is > not yet part of numpy? Make a small Sphinx project for that: $ easy_install numpydoc $ mkdir foo $ cd foo $ sphinx-quickstart ... $ vi conf.py ... add 'sphinx.ext.autodoc', 'numpydoc' to extensions ... $ cp /some/path/modulename.py modulename.py $ vi index.rst ... add .. automodule:: modulename :members: ... $ make PYTHONPATH=$PWD html Could be automated. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Google Groups archive?
Fri, 16 Oct 2009 00:23:34 -0400, josef.pktd wrote: [clip] >> This seems exceedingly odd. Does anyone know _how_ we violated the ToS? > > adult material on front page > > Who's the owner? Creating a new group would require a different name, > since the old name is blocked, I tried. Maybe it's best just not to use Google Groups. IMO, gmane.org offers an equivalent if not superior service. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] object array alignment issues
Fri, 16 Oct 2009 12:07:10 +0200, Francesc Alted wrote: [clip] > IMO, NumPy can be improved for unaligned data handling. For example, > Numexpr is using this small snippet: > > from cpuinfo import cpu > if cpu.is_AMD() or cpu.is_Intel(): > is_cpu_amd_intel = True > else: > is_cpu_amd_intel = False > > for detecting AMD/Intel architectures and allowing the code to avoid > memcpy() calls for the unaligned arrays. > > The above code uses the excellent ``cpuinfo.py`` module from Pearu > Peterson, which is distributed under NumPy, so it should not be too > difficult to take advantage of this for avoiding unnecessary copies in > this scenario. I suppose this kind of check is easiest to do at compile-time, and defining a -DFORCE_ALIGNED? This wouldn't cause performance penalties for those architectures for which they are not necessary. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Objected-oriented SIMD API for Numpy
Wed, 21 Oct 2009 16:48:22 +0900, Mathieu Blondel wrote: [clip] > My original idea was to write the code in C with Intel/Alvitec/Neon > intrinsics and have this code binded to be able to call it from Python. > So the SIMD code would be compiled already, ready to be called from > Python. Like you said, there's a risk that the overhead of calling > Python is bigger than the benefit of using SIMD instructions. If it's > worth trying out, an experiment can be made with Vector4f to see if it's > even worth continuing with other types. The overhead is quickly checked for multiplication with numpy arrays of varying size, without SSE: Overhead per iteration (ms): 1.6264549101 Time per array element (ms): 0.000936947636565 Cross-over point:1735.90801303 #-- import numpy as np from scipy import optimize import time import matplotlib.pyplot as plt def main(): data = [] for n in np.unique(np.logspace(0, 5, 20).astype(int)): print n m = 100 reps = 5 times = [] for rep in xrange(reps): x = np.zeros((n,), dtype=np.float_) start = time.time() #-- for k in xrange(m): x *= 1.1 #-- end = time.time() times.append(end - start) t = min(times) data.append((n, t)) data = np.array(data) def model(z): n, t = data.T overhead, per_elem = z return np.log10(t) - np.log10(overhead + per_elem * n) z, ier = optimize.leastsq(model, [1., 1.]) overhead, per_elem = z print "" print "Overhead per iteration (ms):", overhead*1e3 print "Time per array element (ms):", per_elem*1e3 print "Cross-over point: ", overhead/per_elem n = np.logspace(0, 5, 500) plt.loglog(data[:,0], data[:,0]/data[:,1], 'x', label=r'measured') plt.loglog(n, n/(overhead + per_elem*n), 'k-', label=r'fit to $t = a + b n$') plt.xlabel(r'$n$') plt.ylabel(r'ops/second') plt.grid(1) plt.legend() plt.show() if __name__ == "__main__": main() ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Objected-oriented SIMD API for Numpy
Wed, 21 Oct 2009 14:47:02 +0200, Francesc Alted wrote: [clip] >> Do you have any interest in adding SIMD to some core numpy >> (transcendental functions). If so, I would try to go back to the >> problem of runtime SSE detection and loading of optimized shared >> library in a cross-platform way - that's something which should be done >> at some point in numpy, and people requiring it would be a good >> incentive. > > I don't personally have a lot of interest implementing this for numpy. > But in case anyone does, I find the next library: > > http://gruntthepeon.free.fr/ssemath/ > > very interesting. Perhaps there could be other (free) > implementations... Optimized transcendental functions could be interesting. For example for tanh, call overhead is overcome already for ~30-element arrays. Since these are ufuncs, I suppose the SSE implementations could just be put in a separate module, which is always compiled. Before importing the module, we could simply check from Python side that the CPU supports the necessary instructions. If everything is OK, the accelerated implementations would then just replace the Numpy routines. This type of project could probably also be started outside Numpy, and just monkey-patch the Numpy routines on import. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy and C99
Fri, 23 Oct 2009 09:21:17 -0400, Darren Dale wrote: > Can we use features of C99 in numpy? For example, can we use "//" style > comments, and C99 for statements "for (int i=0, ...) "? It would be much easier if we could, but so far we have strived for C89 compliance. So I guess the answer is "no". -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpydoc without autosummary
Fri, 23 Oct 2009 09:25:12 -0400, Michael Droettboom wrote: > Is there a way to use numpydoc without putting an autosummary table at > the head of each class? I'm using numpydoc primarily for the > sectionized docstring support, but the autosummaries are somewhat > overkill for my project. Numpydoc hooks into sphinx.ext.autodoc's docstring mangling. So if you just need to have docstrings formatted, you can use Sphinx's auto*:: directives. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpydoc without autosummary
Fri, 23 Oct 2009 09:25:12 -0400, Michael Droettboom wrote: > Is there a way to use numpydoc without putting an autosummary table at > the head of each class? I'm using numpydoc primarily for the > sectionized docstring support, but the autosummaries are somewhat > overkill for my project. Ah, you meant the stuff output by default to class docstrings. Currently, there's no way to turn this off, unfortunately. It seems there should be, though... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 500 internal server error from docs.scipy.org
Mon, 26 Oct 2009 08:15:51 -0400, Neal Becker wrote: > This link: > > http://docs.scipy.org/doc/scipy/reference/generated/ scipy.stats.var.html#scipy.stats.var > > gives 500 internal server error Now that's strange. It's a static page. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C code coverage tool
Mon, 26 Oct 2009 14:26:20 -0400, Michael Droettboom wrote: > I know David Cournapeau has done some work on using gcov for coverage > with Numpy. > > Unaware of this, (doh! -- I should have Googled first), I wrote a small > C code-coverage tool built on top of valgrind's callgrind tool, so it > basically only works on x86/AMD64 unixy platforms, but unlike gcov it > doesn't require any recompilation headaches (though compiling > unoptimized helps). [clip] Where's the code? [clip] > Is this something we want to add to the SVN tree, maybe under tools? Yes. Also, maybe you want to send it to the Valgrind guys, too. If they don't yet have a code coverage functionality yet, it could be nice to have. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Segfault when using scipy.special.hermite?
ke, 2009-10-28 kello 14:21 +0100, Ole Streicher kirjoitti: > Is there something wrong with scipy.special.hermite? The following > code produces glibc errors: It's probably this issue: http://projects.scipy.org/numpy/ticket/1211 The most likely cause is that the linear algebra libraries (ATLAS/BLAS/LAPACK) shipped with that version of 64-bit Opensuse are somehow broken. At least on Mandriva it turned out that the problem did not appear if ATLAS was not installed, and it also went away with a newer version of LAPACK. (special.hermite is pure-python code. The only part that can cause problems is scipy.linalg.eig or numpy.linalg.eig, and, much less likely, scipy.special.gamma. The former are thin wrappers around LAPACK routines.) -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] [RFC] complex functions in npymath
Hi (esp. David), If there are no objections, I'd like to move Numpy's complex-valued C99-like functions to npymath: http://github.com/pv/numpy-work/tree/npymath-complex This'll come useful if we want to start eg. writing Ufuncs in Cython. I'm working around possible compiler-incompatibilities of struct return values by having only pointer versions of the functions in libnpymath.a, and the non-pointer versions as inlined static functions. Also, perhaps we should add a header file npy_math_c99compat.h that would detect if the compiler supports C99, and if not, substitute the C99 functions with our npy_math implementations. This'd be great for scipy.special. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [RFC] complex functions in npymath
Fri, 30 Oct 2009 18:34:07 +0900, David Cournapeau wrote: [clip] > Actually, I am in the process of cleaning my numpy branches for review, > and intend to push them into svn as fast as possible. Complex is pretty > high on the list. Great! > The missing piece in complex support in npymath is mostly tests: I have > tests for all the special cases (all special cases specified in C99 > standard are tested), but no test for the actual 'normal' values. If you > (or someone else) could add a couple of tests, that would be great. I can probably take a shot at this. >> I'm working around possible compiler-incompatibilities of struct return >> values by having only pointer versions of the functions in >> libnpymath.a, and the non-pointer versions as inlined static functions. > > Is this a problem if we guarantee that our complex type is bit > compatible with C99 complex (e.g. casting a complex to a double[2] > should alway work) ? > > That's how the complex math is implemented ATM. Correct me if I'm wrong, but I think the problem is that for typedef struct foo foo_t; foo_t bar(); different compilers may put the return value of bar() to a different place (registers vs. memory). If we put those functions in a library, and switch compilers, I think the behavior is undefined as there seems to be no standard. I don't think functions in C can return arrays, so double[2] representation probably does not help us here. >> Also, perhaps we should add a header file >> >> npy_math_c99compat.h >> >> that would detect if the compiler supports C99, and if not, substitute >> the C99 functions with our npy_math implementations. This'd be great >> for scipy.special. > > I am not sure I understand this: currently, if a given complex function > is detected on the platform, npy_foo is just an alias to foo, so we use > the platform implementation whenever possible. I'd like to write code like this: coshf(a) + sinhf(b) and not like this: npy_coshf(a) + npy_sinhf(b) This seems easy to achieve with a convenience header that substitutes the C99 functions with npy_ functions when C99 is not available. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [RFC] complex functions in npymath
pe, 2009-10-30 kello 18:57 +0900, David Cournapeau kirjoitti: [clip: struct return values] > Is this a problem in practice ? If two compilers differ in this, > wouldn't they have incompatible ABI ? Yep, it would be an incompatible ABI. I don't really know how common this in practice -- but there was a comment warning about this in the old ufunc sources, so I wanted to be wary... I don't think there's a significant downside in having thin wrappers around the pointer functions. Googling a bit reveals at least some issues that have cropped up in gcc: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=36834 (MSVC vs. gcc) http://gcc.gnu.org/bugzilla/show_bug.cgi?id=9506 (bug on freebsd) I'd imagine the situation vs. compilers is here a bit similar to C++ ABIs and sounds like it's a less tested corner of the calling conventions. No idea whether this matters in practice, but at least the above MSVC vs. gcc issue sounds like it might bite. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 1.4.0: Setting a firm release date for 1st december.
Mon, 02 Nov 2009 17:29:18 +0900, David Cournapeau wrote: > I think it is about time to release 1.4.0. Instead of proposing a > release date, I am setting a firm date for 1st December, and 16th > november to freeze the trunk. If someone wants a different date, you > have to speak now. > > There are a few issues I would like to clear up: > - Documentation for datetime, in particular for the public C API > - Snow Leopard issues, if any > > Otherwise, I think there has been quite a lot of new features. If people > want to add new functionalities or features, please do it soon, Can we get the complex functions to npy_math for 1.4.0: could be useful for the next Scipy? This is pretty quick to do, I can just write up some more tests one evening and commit. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] persistent ImportError: No module named multiarray when moving cPickle files between machines
Thu, 05 Nov 2009 12:11:32 -0800, Reckoner wrote: > Pauli : > > Thanks for your reply. Using 'wb' instead of 'w' for the file mode > evidently makes this problem go away. > > The mystery is that why does this work with plain lists *and* 'w' and > not with numpy arrays and 'w'. In other words, why is it that > numpy.array needs the binary mode while list does not? Requiring binary is not specific to numpy, but just probably the fact that when you have \r\n linefeeds, pickle ends up trying to load a module called "numpy.core.multiarray\r" which does not exists. Same thing happens with any other types -- it's just that lists are builtin types and don't need any modules loaded to unpickle properly. > Also, for my own edification, how did you know that my "pickle files > produced on Windows were contaminated by \r\n line feeds". I looked in the two files you attached on Numpy trac, and noticed that they were different (sha1sum). -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Deprecate np.max/np.min ?
la, 2009-11-07 kello 18:27 +, Neil Crighton kirjoitti: [clip] > I think it would be better to fix this issue. np.min(3,2) should also give > "ValueError: axis(=2) out of bounds". Fixing this also removes any possibility > of generating hard-to-find errors by overwriting the builtin min/max. (Unless > there's some corner case I'm missing). Fixed in r7697, r7698, which change the behavior of scalars in many functions using axis=xxx. I don't think this breaks anyone's code -- using out-of-bounds values of axis is almost certainly an error. I left axis=-1 and axis=0 allowed, in addition to axis=None. These seemed to be required by at least the masked arrays unit tests... -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] initializing an array of lists
Sat, 07 Nov 2009 21:56:29 -0600, alan wrote: > I want to build a 2D array of lists, and so I need to initialize the > array with empty lists : > > myarray = array([[[],[],[]] ,[[],[],[]]]) > > Is there a clever way to do this? I could define the array > > myarray = zeros( (xdim,ydim), dtype=object) and then iterate through the > elements initializing then to empty lists, but surely there is a better > way. This question seems to come up from time to time (= maybe should be a FAQ?). You can for example vectorize the list constructor: >>> filler = np.frompyfunc(lambda x: list(), 1, 1) >>> a = np.empty((3, 4), dtype=np.object) >>> filler(a, a); array([[[], [], [], []], [[], [], [], []], [[], [], [], []]], dtype=object) >>> a[0,3].append(9) >>> a array([[[], [], [], [9]], [[], [], [], []], [[], [], [], []]], dtype=object) -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Release notes for arraysetops changes
ma, 2009-11-09 kello 23:13 +, Neil Crighton kirjoitti: > I've written some release notes (below) describing the changes to > arraysetops.py. If someone with commit access could check that these sound ok > and add them to the release notes file, that would be great. Thanks, added! Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] allclose() does not check shape of inputs
Fri, 13 Nov 2009 11:54:51 +0100, Robert Cimrman wrote: > I think this is a bug: > > In [16]: np.allclose([1.0, 1.0], [1.1], rtol=0.1, atol=0.0) > Out[16]: True It's broadcasting. I'm not sure it is a bug: >>> np.allclose([1.0, 1.0], [1.1, 1.1, 1.1], rtol=0.1, atol=0.0) False -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] bus error in embedded numpy
Fri, 13 Nov 2009 17:23:19 +, Robin wrote: > I'm trying to embed Python in a MATLAB mex file. I've been coming under > some pressure to make my Python code available to my MATLAB colleagues > so I am trying to come up with a relatively general way of calling > numerical python code from Matlab. Out of curiosity, how are you handling the Matlab RTLD_GLOBAL issue. Last time [1] I did something similar, I had to hack around it in an ugly way. .. [1] http://www.iki.fi/pav/software/pythoncall/index.html -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] bus error in embedded numpy
Fri, 13 Nov 2009 19:12:26 +, Robin wrote: [clip] > How does the RTLD_GLOBAL thing manifest itself? So far I have only a > very basic start which basically consists of: > cmd = mxArrayToString(prhs[0]); > PyRun_SimpleString(cmd); > but I haven't noticed anything not working - I can run numpy testsuite, > and do simple commands as expected (initiliase arrays in the > interpreter, run numpy functions on them). Perhaps recent versions of > Matlab behave differently (I am using R2009a on a mac). The RTLD_GLOBAL issue prevented Numpy from being imported. I think everything was well, until you tried to run "import numpy" in the embedded process -- loading multiarray.so would fail because of missing symbols. But if it worked for you without the hack, then it must have been changed in the Matlab versions since then (and Pythoncall needs updating...). -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] bus error in embedded numpy
pe, 2009-11-13 kello 22:36 +0100, Sturla Molden kirjoitti: [clip] > > I was aware of this - I thought I would be OK on the mac - at least > > Python and Numpy and my mex function are built with apple gcc although > > I'm not sure about Matlab. I guess Windows will be more difficult... > > But in any case I don't plan to pass any file handles around. > > It applies to any CRT resource, not just files. Compiler is not > important, but which CRT is loaded. And if you statically link the same > CRT twice, that becomes two CRTs that cannot share resources. In > Windows, Microsoft has made sure there are multiple versions of their > CRT (msvcrt.dll, msvcr71.dll, msvcr80.dll, msvcr90.dll, ...) that cannot > share resources. And anyone not careful with this can experice crashes > at random locations. Well, for Matlab/Python integration meant for numerical computations I would be surprised if CRT is really a problem. You essentially can pass data to Matlab only via Matlab's own interface -- CRT stuff like ownership of pointers to allocated memory, file handles etc. typically do not cross this boundary. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Failure building pdf doc
Fri, 20 Nov 2009 10:55:35 +0900, David Cournapeau wrote: > While checking everything builds for the 1.4.0 release, I noticed a > problem with building the latex version: > > writing... done > processing numpy-user.tex... user/index user/introduction > user/whatisnumpy user/install user/howtofind user/basics > user/basics.types user/basics.creation user/basics.io > user/basics.io.genfromtxt user/basics.indexing user/basics.broadcasting > user/basics.byteswapping user/basics.rec user/basics.subclassing > user/performance user/misc user/c-info user/c-info.how-to-extend > user/c-info.python-as-glue user/c-info.beyond-basics > resolving references... > writing... > Markup is unsupported in LaTeX: > user/basics.io.genfromtxt:: column or row spanning cells are not yet > implemented. > > How can I fix this ? I noticed a similar bug reported on sphinx bug > tracker 10 months ago by Pauli, is this indeed related ? I think I fixed this in r7751. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 2to3c: 2to3 for C extensions
la, 2009-11-21 kello 20:00 -0500, David Warde-Farley kirjoitti: > Guido posted a link to this on Twitter, it might be of interest to > people interested in the NumPy Python 3 transition: > > http://dmalcolm.livejournal.com/3935.html > > It's doubtful this is a magic bullet (at least, not yet) but it may > still help reduce the amount of busywork involved. Interesting -- though at the moment it doesn't seem to solve very many of our problems. I did some cleanup of numpy.testing, distutils, and core so that using 2to3 the build_ext in numpy.core can at least be started: http://github.com/pv/numpy-work/tree/py3k python3 tools/py3test testing distutils core Of course, the build currently fails with zillion errors in multiarraymodule. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] 2to3c: 2to3 for C extensions
su, 2009-11-22 kello 19:28 +0100, René Dudfield kirjoitti: [clip] > I've been working on this too... see the previous py3k thread for > details of my plan. See http://github.com/illume/numpy3k/tree/work > for my (not very much) progress. > > I don't have time for about a week to do any more work on my tree. > Maybe your tree is ahead anyway... I am taking the approach we used > with pygame so that the tree can work with both python 2.x and 3.x. I > got up to getting a fair amount of the setup code working... but there > are still some issues. Ok, I forgot that you did also some work on this. I think my tree is currently a bit more along the way than yours -- it seems to have the same fixes as in yours, and it proceeds until it fails to compile multiarraymodule. The main difference seems to be that I'm using the 2to3 tool to automatically convert the Python code before the actual build -- this could be a major manual work saver. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Numpy on Python3
http://github.com/pv/numpy-work/tree/py3k $ mkdir -p $PWD/dist/lib/python3.1/site-packages $ python3 setup.py install --prefix=$PWD/dist $ cd $PWD/dist/lib/python3.1/site-packages && python3 Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K: numpy.ma is disabled for now -- some issues >>> numpy.array([1., 2, 3, 4]) array([ 1., 2., 3., 4.]) >>> _ + 10 array([ 11., 12., 13., 14.]) >>> numpy.ones((4,), dtype=complex)/4 array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) >>> numpy.array([object(), object()]) array([, ], dtype=b'object') *** Things were fairly straightforward this far, just many tiny changes. What's left is then sorting out the bigger problems :) This is still far from being complete: - Most use of PyString_* needs auditing (note e.g. the b'object' in the dtype print above...). I simply added convenience wrappers for PyString -> PyBytes, but this is not the correct choice at all points. - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. - Whether to inherit Numpy ints from PyLong_* needs some thinking, as they are quite different objects. Now, I dropped the inheritance, but I wonder if this will break something. - PyFile_AsFile has disappeared, and AsFileDescriptor+fdopen doesn't seem to cut it -- don't know exactly what's wrong here. - Integer -> String formatting does not seem to work - Larger-than-long-long Python ints probably cause problems - The new buffer interface needs to be implemented -- currently there are just error-raising stubs. I remember Dag was working on this a bit: how far did it go? - Relative imports + 2to3 is a bit of a pain. A pity we can't have them in the mainline code because of python2.4. - I didn't check for semantic changes in tp_* interface functions. This we need still to do. - And probably many other issues lurking. Also, I didn't yet try checking how far the test suite passes on Python3. (It still passes completely on Python2, so at least I didn't break that part.) It might be nice to have this merged in at some point after 1.4.0 (after the most obvious glaring bugs have been fixed), so that we could perhaps start aiming for Python3 compatibility in Numpy 1.5.0. Cheers, Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Setup.py runs 2to3 automatically for all changed files. Of course, if it's possible to cater for24 and 3 at the same time,that's good. How do you work around the relative imports andthe changed exception catching syntax? -- alkuper. viesti -- Aihe: Re: [Numpy-discussion] Numpy on Python3 Lähettäjä: David Cournapeau Päivämäärä: 23.11.2009 08:19 On Mon, Nov 23, 2009 at 2:35 PM, Pauli Virtanen wrote: > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. One thing I have on my end is a numpy.distutils which runs under both python 2 and 3, so that you don't have to run 2to3 everytime you make a change. I did not put it in the trunk because I did not want to modify numpy.distutils for 1.4.0 at this point, but I will include the changes as soon as I branch the trunk into 1.4.0, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
The issue with longs is that we wouldn't want array([1,2,3]) create object arrays -- so we need to decide on casting rules for longs. Currently, I think they're treated like python2 ints. -- alkuper. viesti -- Aihe: Re: [Numpy-discussion] Numpy on Python3 Lähettäjä: Charles R Harris Päivämäärä: 23.11.2009 08:08 On Sun, Nov 22, 2009 at 10:35 PM, Pauli Virtanen wrote: > http://github.com/pv/numpy-work/tree/py3k > > $ mkdir -p $PWD/dist/lib/python3.1/site-packages > $ python3 setup.py install --prefix=$PWD/dist > $ cd $PWD/dist/lib/python3.1/site-packages && python3 > Python 3.1.1+ (r311:74480, Oct 11 2009, 20:22:16) > [GCC 4.4.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > XXX: 3K: numpy.random is disabled for now, uses PyString_* > XXX: 3K: numpy.ma is disabled for now -- some issues > >>> numpy.array([1., 2, 3, 4]) > array([ 1., 2., 3., 4.]) > >>> _ + 10 > array([ 11., 12., 13., 14.]) > >>> numpy.ones((4,), dtype=complex)/4 > array([ 0.25+0.j, 0.25+0.j, 0.25+0.j, 0.25+0.j]) > >>> numpy.array([object(), object()]) > array([, ], > dtype=b'object') > >*** > > Things were fairly straightforward this far, just many tiny changes. > What's left is then sorting out the bigger problems :) > > This is still far from being complete: > > - Most use of PyString_* needs auditing (note e.g. the b'object' in the > dtype print above...). > > I simply added convenience wrappers for PyString -> PyBytes, > but this is not the correct choice at all points. > > - Also, should dtype='S' be Bytes or Unicode? I chose Bytes for now. > > - Whether to inherit Numpy ints from PyLong_* needs some thinking, > as they are quite different objects. Now, I dropped the inheritance, > but I wonder if this will break something. > > Maybe. But it was always a hassle because it behaved differently than the other integer types. Now onto float ;) > - PyFile_AsFile has disappeared, and AsFileDescriptor+fdopen doesn't > seem to cut it -- don't know exactly what's wrong here. > > - Integer -> String formatting does not seem to work > > - Larger-than-long-long Python ints probably cause problems > > We used a python call which would raise an error if the number was too large. If that call is still valid, things should work. > - The new buffer interface needs to be implemented -- currently there > are just error-raising stubs. > > I remember Dag was working on this a bit: how far did it go? > > - Relative imports + 2to3 is a bit of a pain. A pity we can't have > them in the mainline code because of python2.4. > > - I didn't check for semantic changes in tp_* interface functions. > This we need still to do. > > I left some notes in the src folder. If you discover anything new put it in there. > - And probably many other issues lurking. > > We do need to look at the initialization of the type math stuff in the ufuncobject module. Yeah, its a bit of a circular dependency, one reason it would be nice to have ufuncs operate on buffer objects instead of ndarrays would be to break the mutual dependence. > > Also, I didn't yet try checking how far the test suite passes on > Python3. (It still passes completely on Python2, so at least I didn't > break that part.) > > It might be nice to have this merged in at some point after 1.4.0 (after > the most obvious glaring bugs have been fixed), so that we could perhaps > start aiming for Python3 compatibility in Numpy 1.5.0. > > If you want to see real suffering, look at the programmer notes in the hacked CRU files. Some poor sucker was stuck with fixing up the g*dawful code while also needing to determine what data was in undocumented binary files, some with the same names but containing different data. Folks, don't let that happen to you. The conversion to Py3k is going to be a breeze by comparison. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Mon, 23 Nov 2009 01:40:00 -0500, Pierre GM wrote: [clip] >> XXX: 3K: numpy.ma is disabled for now -- some issues > > What are the issues ? Something resolving which would have taken more than 5 minutes :) Possibly because something that ma depends on is currently broken in numpy.core. I just wanted to breeze through and arrive as fast as possible at something that can be imported and works for simple things, so I didn't stop at anything that would take longer. I'll take a look at the rest later on. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy on Python3
Mon, 23 Nov 2009 08:58:47 +0100, Sturla Molden wrote: > Pauli Virtanen skrev: >> XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K: >> numpy.ma is disabled for now -- some issues >> > I thought numpy.random uses Cython? Is it just a matter of recompiling > the pyx-file? The Cython file uses the C-api directly, so we'll need a .h file with the necessary compile-time conditionals. >> I remember Dag was working on this a bit: how far did it go? > > Cython's include file numpy.pxd has an ndarray class that extend > PyArrayObject with PEP 3118 buffer compatibility. Great! I believe I will just steal whatever I can and rewrite it in C -- for now, it seems possible to keep Numpy's core in plain C. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Bug in rec.fromarrays ; plus one other possible bug
ke, 2009-11-25 kello 09:48 -0500, Dan Yamins kirjoitti:
> Hi, I'm writing to report what looks like a two bugs in the handling
> of strings of length 0. (I'm using 1.4.0.dev7746, on Mac OSX 10.5.8.
> The problems below occur both for python 2.5 compiled 32-bit as well
> as python2.6 compiled 64-bit).
[clip]
> >>> Cols = [['test']*10,['']*10]
>
> When not passing any dtype, this is created into a recarray with no
> problem:
>
> >>> np.rec.fromarrays(Cols)
> rec.array([('test', ''), ('test', ''), ('test', ''), ('test', ''),
>('test', ''), ('test', ''), ('test', ''), ('test', ''),
>('test', ''), ('test', '')],
> dtype=[('f0', '|S4'), ('f1', '|S1')])
>
> However, trouble arises when I try to pass a length-0 dtype
> explicitly.
>
> >>> d = np.dtype([('A', '|S4'), ('B', '|S')])
> >>> np.rec.fromarrays(Cols,dtype=d)
> recarray([('test', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est',
> ''),
>('\x00est', ''), ('\x00est', ''), ('\x00est', ''), ('\x00est',
> ''),
>('\x00est', ''), ('\x00est', '')],
> dtype=[('A', '|S4'), ('B', '|S0')])
That certainly looks like a bug -- where does the \0 appear in front of
all but the first string?
File a ticket in Trac, http://projects.scipy.org/numpy/ -- small bugs
like this tend to get lost and forgotten in mailing list traffic.
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Correlation function about a factor of 100 slower than matlab/mathcad ... but with fft even worse ?
ke, 2009-11-25 kello 19:23 +0100, qu...@gmx.at kirjoitti: [clip] > Could someone please investigate why correlate and especially > fftconvolve are orders of magnitude slower? Read http://projects.scipy.org/numpy/ticket/1260 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Bytes vs. Unicode in Python3
Hi,
The Python 3 porting needs some decisions on what is Bytes and
what is Unicode.
I'm currently taking the following approach. Comments?
***
dtype field names
Either Bytes or Unicode.
But 'a' and b'a' are *different* fields.
The issue is that:
Python 2: {'a': 2}[u'a'] == 2, {u'a': 2}['a'] == 2
Python 3: {'a': 2}[b'a'], {b'a': 2}['a'] raise exceptions
so the current assumptions in the C code of u'a' == b'a'
cease to hold.
dtype titles
If Bytes or Unicode, work similarly as field names.
dtype format strings, datetime tuple, and any other "protocol" strings
Bytes. User can pass in Unicode, but it's converted using
UTF8 codec.
This will likely change repr() of various objects. Acceptable?
--
Pauli Virtanen
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
