Re: [Numpy-discussion] Proposed Roadmap Overview
On 17.02.2012, at 21:46, Ralf Gommers wrote: [...] So far no one has managed to build the numpy/scipy combo with the LLVM-based compilers, so if you were willing to have a go at fixing that it would be hugely appreciated. See http://projects.scipy.org/scipy/ticket/1500 for details. Once that's fixed, numpy can switch to using it for releases. Well, I had great success with using clang and clang++ (which uses llvm) to compile both numpy and scipy on OS X 10.7.3. Samuel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
20.02.2012 08:35, Paul Anton Letnes kirjoitti: In the language wars, I have one question. Why is Fortran not being considered? Fortran is OK for simple numerical algorithms, but starts to suck heavily if you need to do any string handling, I/O, complicated logic, or data structures. Most of the work in Numpy implementation is not actually in numerics, but in figuring out the correct operation to dispatch the computations to. So, this is one reason why Fortran is not considered. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Mon, Feb 20, 2012 at 1:54 AM, Pauli Virtanen p...@iki.fi wrote: 20.02.2012 08:35, Paul Anton Letnes kirjoitti: In the language wars, I have one question. Why is Fortran not being considered? Fortran is OK for simple numerical algorithms, but starts to suck heavily if you need to do any string handling, I/O, complicated logic, or data structures. Out of curiosity, is this still true for the latest Fortran versions? I guess there the problem may be compiler support over various platforms. Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Mon, Feb 20, 2012 at 2:54 AM, Pauli Virtanen p...@iki.fi wrote: 20.02.2012 08:35, Paul Anton Letnes kirjoitti: In the language wars, I have one question. Why is Fortran not being considered? Fortran is OK for simple numerical algorithms, but starts to suck heavily if you need to do any string handling, I/O, complicated logic, or data structures. Most of the work in Numpy implementation is not actually in numerics, but in figuring out the correct operation to dispatch the computations to. So, this is one reason why Fortran is not considered. There also used to be a problem with unsigned types not being available. I don't know if that is still the case. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 12:43, skrev Charles R Harris: There also used to be a problem with unsigned types not being available. I don't know if that is still the case. Fortran -- like Python and Java -- does not have built-in unsigned integer types. It is never really a problem though. One can e.g. use a longer integer or keep them in an array of bytes. (Fortran 2003 is OOP so it is possible to define one if needed. Not saying it is a good idea.) Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 10:54, skrev Pauli Virtanen: Fortran is OK for simple numerical algorithms, but starts to suck heavily if you need to do any string handling, I/O, complicated logic, or data structures For string handling, C is actually worse than Fortran. In Fortran a string can be sliced like in Python. It is not as nice as Python, but far better than C. Fortran's built-in I/O syntax is archaic, but the ISO C bindings in Fortran 2003 means one can use other means of I/O (posix, win api, C stdio) in a portable way. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 08:35, skrev Paul Anton Letnes: In the language wars, I have one question. Why is Fortran not being considered? Fortran already implements many of the features that we want in NumPy: Yes ... but it does not make Fortran a systems programming language. Making NumPy is different from using it. - slicing and similar operations, at least some of the fancy indexing kind - element-wise array operations and function calls - array bounds-checking and other debugging aid (with debugging flags) That is nice for numerical computing, but not really needed to make NumPy. - arrays that mentally map very well onto numpy arrays. To me, this spells +1 to ease of contribution, over some abstract C/C++ template Mentally perhaps, but not binary. NumPy needs uniformly strided memory on the binary level. Fortran just gives this at the mental level. E.g. there is nothing that dictates a Fortran pointer has to be a view, the compiler is free to employ copy-in copy-out. In Fortran, a function call can invalidate a pointer. One would therefore have to store the array in an array of integer*1, and use the intrinsic function transfer() to parse the contents into NumPy dtypes. - in newer standards it has some nontrivial mathematical functions: gamma, bessel, etc. that numpy lacks right now That belongs to SciPy. - compilers that are good at optimizing for floating-point performance, because that's what Fortran is all about Insanely good, but not when we start to do the (binary, not mentally) strided access that NumPy needs. (Not that C compilers would be any better.) - not Fortran as such, but BLAS and LAPACK are easily accessed by Fortran - possibly other numerical libraries that can be helpful - Fortran has, in its newer standards, thought of C interoperability. We could still keep bits of the code in C (or even C++?) if we'd like to, or perhaps f2py/Cython could do the wrapping. Not f2py, as it depends on NumPy. - some programmers know Fortran better than C++. Fortran is at least used by many science guys, like me. That is a valid arguments. Fortran is also much easier to read and debug. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 08:35, skrev Paul Anton Letnes: As far as I can understand, implementing element-wise operations, slicing, and a host of other NumPy features is in some sense pointless - the Fortran compiler authors have already done it for us. Only if you know the array dimensions in advance. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 19.02.2012 00:09, skrev David Cournapeau: There are better languages than C++ that has most of the technical benefits stated in this discussion (rust and D being the most obvious ones), but whose usage is unrealistic today for various reasons: knowledge, availability on esoteric platforms, etc… A new language is completely ridiculous. There are still other options than C or C++ that are worth considering. One would be to write NumPy in Python. E.g. we could use LLVM as a JIT-compiler and produce the performance critical code we need on the fly. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 17:42, skrev Sturla Molden: There are still other options than C or C++ that are worth considering. One would be to write NumPy in Python. E.g. we could use LLVM as a JIT-compiler and produce the performance critical code we need on the fly. LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster than GCC and often produces better machine code. They can therefore be used inside an array library. It would give a faster NumPy, and we could keep most of it in Python. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Mon, Feb 20, 2012 at 9:55 AM, Sturla Molden stu...@molden.no wrote: Den 20.02.2012 17:42, skrev Sturla Molden: There are still other options than C or C++ that are worth considering. One would be to write NumPy in Python. E.g. we could use LLVM as a JIT-compiler and produce the performance critical code we need on the fly. LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster than GCC and often produces better machine code. They can therefore be used inside an array library. It would give a faster NumPy, and we could keep most of it in Python. Would that work for Ruby also? One of the advantages of C++ is that the code doesn't need to be refactored to start with, just modified step by step going into the future. I think PyPy is close to what you are talking about. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On 02/20/2012 08:55 AM, Sturla Molden wrote: Den 20.02.2012 17:42, skrev Sturla Molden: There are still other options than C or C++ that are worth considering. One would be to write NumPy in Python. E.g. we could use LLVM as a JIT-compiler and produce the performance critical code we need on the fly. LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster than GCC and often produces better machine code. They can therefore be used inside an array library. It would give a faster NumPy, and we could keep most of it in Python. I think it is moot to focus on improving NumPy performance as long as in practice all NumPy operations are memory bound due to the need to take a trip through system memory for almost any operation. C/C++ is simply good enough. JIT is when you're chasing a 2x improvement or so, but today NumPy can be 10-20x slower than a Cython loop. You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting... Dag ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
C++11 has this option:
for (auto item : container) {
// iterate over the container object,
// get a reference to each item
//
// container can be an STL class or
// A C-style array with known size.
}
Which does this:
for item in container:
pass
It is even better than using the macro way because the compiler knows
everything is constant (start and end), so it can do better things.
Using C++ templates to generate ufunc loops is an obvious application,
but again, in the simple examples
Template metaprogramming?
Don't even think about it. It is brain dead to try to outsmart the
compiler.
It is really easy to outsmart the compiler. Really. I use metaprogramming
for loop creation to optimize cache behavior, communication in parallel
environments, and there is no way the compiler would have done things as
efficiently (and there is a lot of leeway to enhance my code).
--
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Would it be fair to say then, that you are expecting the discussion about C++ will mainly arise after the Mark has written the code? I can see that it will be easier to specific at that point, but there must be a serious risk that it will be too late to seriously consider an alternative approach. We will need to see examples of what Mark is talking about and clarify some of the compiler issues. Certainly there is some risk that once code is written that it will be tempting to just use it. Other approaches are certainly worth exploring in the mean-time, but C++ has some strong arguments for it. Compilers for C++98 are now stable enough (except on Bluegene, see the Boost distribution with xlc++) C++ helps a lot to enhance robustness.ts? From my perspective having a standalone core NumPy is still a goal. The primary advantages of having a NumPy library (call it NumLib for the sake of argument) are 1) Ability for projects like PyPy, IronPython, and Jython to use it more easily 2) Ability for Ruby, Perl, Node.JS, and other new languages to use the code for their technical computing projects. 3) increasing the number of users who can help make it more solid 4) being able to build the user-base (and corresponding performance with eye-balls from Intel, NVidia, AMD, Microsoft, Google, etc. looking at the code). The disadvantages I can think of: 1) More users also means we might risk lowest-commond-denominator problems --- i.e. trying to be too much to too many may make it not useful for anyone. Also, more users means more people with opinions that might be difficult to re-concile. 2) The work of doing the re-write is not small: probably at least 6 person-months 3) Not being able to rely on Python objects (dictionaries, lists, and tuples are currently used in the code-base quite a bit --- though the re-factor did show some examples of how to remove this usage). 4) Handling of Object arrays requires some re-design. I'm sure there are other factors that could be added to both lists. -Travis Thanks a lot for the reply, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
2012/2/19 Matthew Brett matthew.br...@gmail.com Hi, On Sat, Feb 18, 2012 at 8:38 PM, Travis Oliphant tra...@continuum.io wrote: We will need to see examples of what Mark is talking about and clarify some of the compiler issues. Certainly there is some risk that once code is written that it will be tempting to just use it. Other approaches are certainly worth exploring in the mean-time, but C++ has some strong arguments for it. The worry as I understand it is that a C++ rewrite might make the numpy core effectively a read-only project for anyone but Mark. Do you have any feeling for whether that is likely? Some of us are C developers, other are C++. It will depend on the background of each of us. How would numpylib compare to libraries like eigen? How likely do you think it would be that unrelated projects would use numpylib rather than eigen or other numerical libraries? Do you think the choice of C++ rather than C will influence whether other projects will take it up? I guess that the C++ port may open a door to change the back-end, and perhaps use Eigen, or ArBB. As those guys (ArBB) wanted to provided a Python interface compatible with Numpy to their VM, it may be interesting to be able to change back-ends (although it is limited to one platform and 2 OS). -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 18:14, skrev Charles R Harris: Would that work for Ruby also? One of the advantages of C++ is that the code doesn't need to be refactored to start with, just modified step by step going into the future. I think PyPy is close to what you are talking about. If we plant to support more languages than Python, it might be better to use C++ (sorry). But it does not mean that LLVM cannot be used. Either one can generate C or C++, or just use the assembly language (which is very simple and readable too: http://llvm.org/docs/LangRef.html). We have exact knowledge about an ndarray at runtime: - dtype - dimensions - strides - whether the array is contiguous or not This can be JIT-compiled into specialized looping code by LLVM. These kernels can then be stored in a database and resued. If it matters, LLVM is embeddable in C++. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
2012/2/19 Nathaniel Smith n...@pobox.com On Sun, Feb 19, 2012 at 9:16 AM, David Cournapeau courn...@gmail.com wrote: On Sun, Feb 19, 2012 at 8:08 AM, Mark Wiebe mwwi...@gmail.com wrote: Is there a specific target platform/compiler combination you're thinking of where we can do tests on this? I don't believe the compile times are as bad as many people suspect, can you give some simple examples of things we might do in NumPy you expect to compile slower in C++ vs C? Switching from gcc to g++ on the same codebase should not change much compilation times. We should test, but that's not what worries me. What worries me is when we start using C++ specific code, STL and co. Today, scipy.sparse.sparsetools takes half of the build time of the whole scipy, and it does not even use fancy features. It also takes Gb of ram when building in parallel. I like C++ but it definitely does have issues with compilation times. IIRC the main problem is very simple: STL and friends (e.g. Boost) are huge libraries, and because they use templates, the entire source code is in the header files. That means that as soon as you #include a few standard C++ headers, your innocent little source file has suddenly become hundreds of thousands of lines long, and it just takes the compiler a while to churn through megabytes of source code, no matter what it is. (Effectively you recompile some significant fraction of STL from scratch on every file, and then throw it away.) In fact Boost tries to be clean about this. Up to a few minor releases of GCC, their headers were a mess. When you included something, a lot of additional code was brought, and the compile-time exploded. But this is no longer the case. If we restrict the core to a few includes, even with templates, it should not be long to compile. -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting… Hmm, that would be cute indeed. Do you have an idea on how the code in the with context could be passed to the Python AST compiler (à la numexpr.evaluate(A + B + C))? -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
2012/2/19 Sturla Molden stu...@molden.no Den 19.02.2012 10:28, skrev Mark Wiebe: Particular styles of using templates can cause this, yes. To properly do this kind of advanced C++ library work, it's important to think about the big-O notation behavior of your template instantiations, not just the big-O notation of run-time. C++ templates have a turing-complete language (which is said to be quite similar to haskell, but spelled vastly different) running at compile time in them. This is what gives template meta-programming in C++ great power, but since templates weren't designed for this style of programming originally, template meta-programming is not very easy. The problem with metaprogramming is that we are doing manually the work that belongs to the compiler. Blitz++ was supposed to be a library that thought like a compiler. But then compilers just got better. Today, it is no longer possible for a numerical library programmer to outsmart an optimizing C++ compiler. All metaprogramming can do today is produce error messages noone can understand. And the resulting code will often be slower because the compiler has less opportunities to do its work. As I've said, the compiler is pretty much stupid. It cannot do what Blitzz++ did, or what Eigen is currently doing, mainly because of the basis different languages (C or C++). -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Mon, Feb 20, 2012 at 9:18 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: On 02/20/2012 08:55 AM, Sturla Molden wrote: Den 20.02.2012 17:42, skrev Sturla Molden: There are still other options than C or C++ that are worth considering. One would be to write NumPy in Python. E.g. we could use LLVM as a JIT-compiler and produce the performance critical code we need on the fly. LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster than GCC and often produces better machine code. They can therefore be used inside an array library. It would give a faster NumPy, and we could keep most of it in Python. I think it is moot to focus on improving NumPy performance as long as in practice all NumPy operations are memory bound due to the need to take a trip through system memory for almost any operation. C/C++ is simply good enough. JIT is when you're chasing a 2x improvement or so, but today NumPy can be 10-20x slower than a Cython loop. I don't follow this. Could you expand a bit more? (Specifically, I wasn't aware that numpy could be 10-20x slower than a cython loop, if we're talking about the base numpy library--so core operations. I'm also not totally sure why a JIT is a 2x improvement or so vs. cython. Not that a disagree on either of these points, I'd just like a bit more detail.) Thanks, Chris You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting... Dag ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 18:18, skrev Dag Sverre Seljebotn: I think it is moot to focus on improving NumPy performance as long as in practice all NumPy operations are memory bound due to the need to take a trip through system memory for almost any operation. C/C++ is simply good enough. JIT is when you're chasing a 2x improvement or so, but today NumPy can be 10-20x slower than a Cython loop. You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting... Lazy evaluation is nice. But I was thinking more about how to avoid C++ in the NumPy core, so more than 2 or 3 programmers could contribute. I.e. my point was not that loops in LLVM would be much faster than C++ (that is besides the point), but the code could be written in Python instead of C++. But if the idea is to support other languages as well (which I somehow forgot), then this approach certainly becomes less useful. (OTOH, lazy evaluation is certainly easier to achieve with JIT compilation. But that will have to wait until NumPy 5.0 perhaps...) Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ndarray and lazy evaluation (was: Proposed Rodmap Overview)
On 02/20/2012 09:24 AM, Olivier Delalleau wrote: Hi Dag, Would you mind elaborating a bit on that example you mentioned at the end of your email? I don't quite understand what behavior you would like to achieve Sure, see below. I think we should continue discussion on numpy-discuss. I wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting... More information: The disadvantage today of using Theano (or numexpr) is that they require using a different API, so that one has to learn and use Theano from the ground up, rather than just slap it on in an optimization phase. The alternative would require extensive changes to NumPy, so I guess Theano authors or Francesc would need to push for this. The alternative would be (with A, B, C ndarray instances): with theano.lazy: arr = A + B + C On __enter__, the context manager would hook into NumPy to override it's arithmetic operators. Then it would build a Theano symbolic tree instead of performing computations right away. In addition to providing support for overriding arithmetic operators, slicing etc., it would be necesarry for arr to be an ndarray instance which is not yet computed (data-pointer set to NULL, and store a compute-me callback and some context information). Finally, the __exit__ would trigger computation. For other operations which need the data pointer (e.g., single element lookup) one could either raise an exception or trigger computation. This is just a rough sketch. It is not difficult in principle, but of course there's really a massive amount of work involved to work support for this into the NumPy APIs. Probably, we're talking a NumPy 3.0 thing, after the current round of refactorings have settled... Please: Before discussing this further one should figure out if there's manpower available for it; no sense in hashing out a castle in the sky in details. Also it would be better to talk in person about this if possible (I'm in Berkeley now and will attend PyData and PyCon). Dag ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 18:34, skrev Christopher Jordan-Squire: I don't follow this. Could you expand a bit more? (Specifically, I wasn't aware that numpy could be 10-20x slower than a cython loop, if we're talking about the base numpy library--so core operations. I'm also not totally sure why a JIT is a 2x improvement or so vs. cython. Not that a disagree on either of these points, I'd just like a bit more detail.) Dag Sverre is right about this. NumPy is memory bound, Cython loops are (usually) CPU bound. If you write: x[:] = a + b + c # numpy arrays then this happens (excluding reference counting): - allocate temporary array - loop over a and b, add to temporary - allocate 2nd temporary array - loop over 1st temporary array and c, add to 2nd - deallocate 1st temporary array - loop over 2nd temporary array, assign to x - deallocate 2nd temporary array Since memory access is slow, memory allocation and deallocation is slow, and computation is fast, this will be perhaps 10 times slower than what we could do with a loop in Cython: for i in range(n): x[i] = a[i] + b[i] + c[i] I.e. we get rid of the temporary arrays and the multiple loops. All the temporaries here are put in registers. It is streaming data into the CPU that is slow, not computing! It has actually been experimented with streaming data in a compressed form, and decompressing on the fly, as data access still dominates the runtime (even if you do a lot of computing per element). Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray and lazy evaluation (was: Proposed Rodmap Overview)
On Feb 20, 2012, at 6:46 PM, Dag Sverre Seljebotn wrote: On 02/20/2012 09:24 AM, Olivier Delalleau wrote: Hi Dag, Would you mind elaborating a bit on that example you mentioned at the end of your email? I don't quite understand what behavior you would like to achieve Sure, see below. I think we should continue discussion on numpy-discuss. I wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting... More information: The disadvantage today of using Theano (or numexpr) is that they require using a different API, so that one has to learn and use Theano from the ground up, rather than just slap it on in an optimization phase. The alternative would require extensive changes to NumPy, so I guess Theano authors or Francesc would need to push for this. The alternative would be (with A, B, C ndarray instances): with theano.lazy: arr = A + B + C On __enter__, the context manager would hook into NumPy to override it's arithmetic operators. Then it would build a Theano symbolic tree instead of performing computations right away. In addition to providing support for overriding arithmetic operators, slicing etc., it would be necesarry for arr to be an ndarray instance which is not yet computed (data-pointer set to NULL, and store a compute-me callback and some context information). Finally, the __exit__ would trigger computation. For other operations which need the data pointer (e.g., single element lookup) one could either raise an exception or trigger computation. This is just a rough sketch. It is not difficult in principle, but of course there's really a massive amount of work involved to work support for this into the NumPy APIs. Probably, we're talking a NumPy 3.0 thing, after the current round of refactorings have settled... Please: Before discussing this further one should figure out if there's manpower available for it; no sense in hashing out a castle in the sky in details. I see. Mark Wiebe already suggested the same thing some time ago: https://github.com/numpy/numpy/blob/master/doc/neps/deferred-ufunc-evaluation.rst Also it would be better to talk in person about this if possible (I'm in Berkeley now and will attend PyData and PyCon). Nice. Most of Continuum crew (me included) will be attending to both conferences. Mark W. will make PyCon only, but will be a good occasion to discuss this further. See you, -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray and lazy evaluation (was: Proposed Rodmap Overview)
Never mind. The link Francesc posted answered my question :) -=- Olivier Le 20 février 2012 12:54, Olivier Delalleau delal...@iro.umontreal.ca a écrit : Le 20 février 2012 12:46, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no a écrit : On 02/20/2012 09:24 AM, Olivier Delalleau wrote: Hi Dag, Would you mind elaborating a bit on that example you mentioned at the end of your email? I don't quite understand what behavior you would like to achieve Sure, see below. I think we should continue discussion on numpy-discuss. I wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting... More information: The disadvantage today of using Theano (or numexpr) is that they require using a different API, so that one has to learn and use Theano from the ground up, rather than just slap it on in an optimization phase. The alternative would require extensive changes to NumPy, so I guess Theano authors or Francesc would need to push for this. The alternative would be (with A, B, C ndarray instances): with theano.lazy: arr = A + B + C On __enter__, the context manager would hook into NumPy to override it's arithmetic operators. Then it would build a Theano symbolic tree instead of performing computations right away. In addition to providing support for overriding arithmetic operators, slicing etc., it would be necesarry for arr to be an ndarray instance which is not yet computed (data-pointer set to NULL, and store a compute-me callback and some context information). Finally, the __exit__ would trigger computation. For other operations which need the data pointer (e.g., single element lookup) one could either raise an exception or trigger computation. This is just a rough sketch. It is not difficult in principle, but of course there's really a massive amount of work involved to work support for this into the NumPy APIs. Probably, we're talking a NumPy 3.0 thing, after the current round of refactorings have settled... Please: Before discussing this further one should figure out if there's manpower available for it; no sense in hashing out a castle in the sky in details. Also it would be better to talk in person about this if possible (I'm in Berkeley now and will attend PyData and PyCon). Dag Thanks for the additional details. I feel like this must be a stupid question, but I have to ask: what is the point of being lazy here, since the computation is performed on exit anyway? -=- Olivier ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On 02/20/2012 09:34 AM, Christopher Jordan-Squire wrote: On Mon, Feb 20, 2012 at 9:18 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: On 02/20/2012 08:55 AM, Sturla Molden wrote: Den 20.02.2012 17:42, skrev Sturla Molden: There are still other options than C or C++ that are worth considering. One would be to write NumPy in Python. E.g. we could use LLVM as a JIT-compiler and produce the performance critical code we need on the fly. LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster than GCC and often produces better machine code. They can therefore be used inside an array library. It would give a faster NumPy, and we could keep most of it in Python. I think it is moot to focus on improving NumPy performance as long as in practice all NumPy operations are memory bound due to the need to take a trip through system memory for almost any operation. C/C++ is simply good enough. JIT is when you're chasing a 2x improvement or so, but today NumPy can be 10-20x slower than a Cython loop. I don't follow this. Could you expand a bit more? (Specifically, I wasn't aware that numpy could be 10-20x slower than a cython loop, if we're talking about the base numpy library--so core operations. I'm The problem with NumPy is the temporaries needed -- if you want to compute A + B + np.sqrt(D) then, if the arrays are larger than cache size (a couple of megabytes), then each of those operations will first transfer the data in and out over the memory bus. I.e. first you compute an element of sqrt(D), then the result of that is put in system memory, then later the same number is read back in order to add it to an element in B, and so on. The compute-to-bandwidth ratio of modern CPUs is between 30:1 and 60:1... so in extreme cases it's cheaper to do 60 additions than to transfer a single number from system memory. It is much faster to only transfer an element (or small block) from each of A, B, and D to CPU cache, then do the entire expression, then transfer the result back. This is easy to code in Cython/Fortran/C and impossible with NumPy/Python. This is why numexpr/Theano exists. You can make the slowdown over Cython/Fortran/C almost arbitrarily large by adding terms to the equation above. So of course, the actual slowdown depends on your usecase. also not totally sure why a JIT is a 2x improvement or so vs. cython. Not that a disagree on either of these points, I'd just like a bit more detail.) I meant that the JIT may be a 2x improvement over the current NumPy C code. There's some logic when iterating arrays that could perhaps be specialized away depending on the actual array layout at runtime. But I'm thinking that probably a JIT wouldn't help all that much, so it's probably 1x -- the 2x was just to be very conservative w.r.t. the argument I was making, as I don't know the NumPy C sources well enough. Dag ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray and lazy evaluation
On 02/20/2012 10:04 AM, Francesc Alted wrote: On Feb 20, 2012, at 6:46 PM, Dag Sverre Seljebotn wrote: On 02/20/2012 09:24 AM, Olivier Delalleau wrote: Hi Dag, Would you mind elaborating a bit on that example you mentioned at the end of your email? I don't quite understand what behavior you would like to achieve Sure, see below. I think we should continue discussion on numpy-discuss. I wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting... More information: The disadvantage today of using Theano (or numexpr) is that they require using a different API, so that one has to learn and use Theano from the ground up, rather than just slap it on in an optimization phase. The alternative would require extensive changes to NumPy, so I guess Theano authors or Francesc would need to push for this. The alternative would be (with A, B, C ndarray instances): with theano.lazy: arr = A + B + C On __enter__, the context manager would hook into NumPy to override it's arithmetic operators. Then it would build a Theano symbolic tree instead of performing computations right away. In addition to providing support for overriding arithmetic operators, slicing etc., it would be necesarry for arr to be an ndarray instance which is not yet computed (data-pointer set to NULL, and store a compute-me callback and some context information). Finally, the __exit__ would trigger computation. For other operations which need the data pointer (e.g., single element lookup) one could either raise an exception or trigger computation. This is just a rough sketch. It is not difficult in principle, but of course there's really a massive amount of work involved to work support for this into the NumPy APIs. Probably, we're talking a NumPy 3.0 thing, after the current round of refactorings have settled... Please: Before discussing this further one should figure out if there's manpower available for it; no sense in hashing out a castle in the sky in details. I see. Mark Wiebe already suggested the same thing some time ago: https://github.com/numpy/numpy/blob/master/doc/neps/deferred-ufunc-evaluation.rst Thanks, I didn't know about that (though I did really assume this was on Mark's radar already). Also it would be better to talk in person about this if possible (I'm in Berkeley now and will attend PyData and PyCon). Nice. Most of Continuum crew (me included) will be attending to both conferences. Mark W. will make PyCon only, but will be a good occasion to discuss this further. I certainly don't think I have anything to add to this discussion beyond what Mark wrote up. But will be nice to meet up anyway. Dag ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Feb 20, 2012, at 7:08 PM, Dag Sverre Seljebotn wrote: On 02/20/2012 09:34 AM, Christopher Jordan-Squire wrote: On Mon, Feb 20, 2012 at 9:18 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: On 02/20/2012 08:55 AM, Sturla Molden wrote: Den 20.02.2012 17:42, skrev Sturla Molden: There are still other options than C or C++ that are worth considering. One would be to write NumPy in Python. E.g. we could use LLVM as a JIT-compiler and produce the performance critical code we need on the fly. LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster than GCC and often produces better machine code. They can therefore be used inside an array library. It would give a faster NumPy, and we could keep most of it in Python. I think it is moot to focus on improving NumPy performance as long as in practice all NumPy operations are memory bound due to the need to take a trip through system memory for almost any operation. C/C++ is simply good enough. JIT is when you're chasing a 2x improvement or so, but today NumPy can be 10-20x slower than a Cython loop. I don't follow this. Could you expand a bit more? (Specifically, I wasn't aware that numpy could be 10-20x slower than a cython loop, if we're talking about the base numpy library--so core operations. I'm The problem with NumPy is the temporaries needed -- if you want to compute A + B + np.sqrt(D) then, if the arrays are larger than cache size (a couple of megabytes), then each of those operations will first transfer the data in and out over the memory bus. I.e. first you compute an element of sqrt(D), then the result of that is put in system memory, then later the same number is read back in order to add it to an element in B, and so on. The compute-to-bandwidth ratio of modern CPUs is between 30:1 and 60:1... so in extreme cases it's cheaper to do 60 additions than to transfer a single number from system memory. It is much faster to only transfer an element (or small block) from each of A, B, and D to CPU cache, then do the entire expression, then transfer the result back. This is easy to code in Cython/Fortran/C and impossible with NumPy/Python. This is why numexpr/Theano exists. Well, I can't speak for Theano (it is quite more general than numexpr, and more geared towards using GPUs, right?), but this was certainly the issue that make David Cooke to create numexpr. A more in-deep explanation about this problem can be seen in: http://www.euroscipy.org/talk/1657 which includes some graphical explanations. -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Forbidden charcter in the names argument of genfromtxt?
On Sat, Feb 18, 2012 at 8:12 PM, Adam Hughes hugad...@gwmail.gwu.edu wrote:
Hey everyone,
I have timeseries data in which the column label is simply a filename from
which the original data was taken. Here's some sample data:
name1.txt name2.txt name3.txt
32 34 953
32 03 402
I've noticed that the standard genfromtxt() method works great; however, the
names aren't written correctly. That is, if I use the command:
print data['name1.txt']
Nothing happens.
However, when I remove the file extension, Eg:
name1 name2 name3
32 34 953
32 03 402
Then print data['name1'] return (32, 32) as expected. It seems that the
period in the name isn't compatible with the genfromtxt() names attribute.
Is there a workaround, or do I need to restructure my program to get the
extension removed? I'd rather not do this if possible for reasons that
aren't important for the discussion at hand.
It looks like the period is just getting stripped out of the names:
In [1]: import numpy as N
In [2]: N.genfromtxt('sample.txt', names=True)
Out[2]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1txt', 'f8'), ('name2txt', 'f8'), ('name3txt', 'f8')])
Interestingly, this still happens if you supply the names manually:
In [17]: def reader(filename):
: infile = open(filename, 'r')
: names = infile.readline().split()
: data = N.genfromtxt(infile, names=names)
: infile.close()
: return data
:
In [20]: data = reader('sample.txt')
In [21]: data
Out[21]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1txt', 'f8'), ('name2txt', 'f8'), ('name3txt', 'f8')])
What you can do is reset the names after genfromtxt is through with it, though:
In [34]: def reader(filename):
: infile = open(filename, 'r')
: names = infile.readline().split()
: infile.close()
: data = N.genfromtxt(filename, names=True)
: data.dtype.names = names
: return data
:
In [35]: data = reader('sample.txt')
In [36]: data
Out[36]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1.txt', 'f8'), ('name2.txt', 'f8'), ('name3.txt', 'f8')])
Be warned, I don't know why the period is getting stripped; there may
be a good reason, and adding it in might cause problems.
~Brett
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Forbidden charcter in the names argument of genfromtxt?
Thanks Brett. I appreciate you taking the time to help me out. In
particular, I did not know the correct syntax for this:
data.dtype.names = names
Which is very helpful. If I would have known how to access
data.dtype.names, I think it would have saved me a great deal of trouble.
I guess it's all part of a learning curve. I'll keep in mind that the
period may cause problems later; however, as far as I can tell so far,
there's nothing going wrong when I access the data.
On Mon, Feb 20, 2012 at 1:35 PM, Brett Olsen brett.ol...@gmail.com wrote:
On Sat, Feb 18, 2012 at 8:12 PM, Adam Hughes hugad...@gwmail.gwu.edu
wrote:
Hey everyone,
I have timeseries data in which the column label is simply a filename
from
which the original data was taken. Here's some sample data:
name1.txt name2.txt name3.txt
32 34953
32 03402
I've noticed that the standard genfromtxt() method works great; however,
the
names aren't written correctly. That is, if I use the command:
print data['name1.txt']
Nothing happens.
However, when I remove the file extension, Eg:
name1 name2 name3
32 34953
32 03402
Then print data['name1'] return (32, 32) as expected. It seems that the
period in the name isn't compatible with the genfromtxt() names
attribute.
Is there a workaround, or do I need to restructure my program to get the
extension removed? I'd rather not do this if possible for reasons that
aren't important for the discussion at hand.
It looks like the period is just getting stripped out of the names:
In [1]: import numpy as N
In [2]: N.genfromtxt('sample.txt', names=True)
Out[2]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1txt', 'f8'), ('name2txt', 'f8'), ('name3txt', 'f8')])
Interestingly, this still happens if you supply the names manually:
In [17]: def reader(filename):
: infile = open(filename, 'r')
: names = infile.readline().split()
: data = N.genfromtxt(infile, names=names)
: infile.close()
: return data
:
In [20]: data = reader('sample.txt')
In [21]: data
Out[21]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1txt', 'f8'), ('name2txt', 'f8'), ('name3txt', 'f8')])
What you can do is reset the names after genfromtxt is through with it,
though:
In [34]: def reader(filename):
: infile = open(filename, 'r')
: names = infile.readline().split()
: infile.close()
: data = N.genfromtxt(filename, names=True)
: data.dtype.names = names
: return data
:
In [35]: data = reader('sample.txt')
In [36]: data
Out[36]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1.txt', 'f8'), ('name2.txt', 'f8'), ('name3.txt',
'f8')])
Be warned, I don't know why the period is getting stripped; there may
be a good reason, and adding it in might cause problems.
~Brett
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Forbidden charcter in the names argument of genfromtxt?
On Mon, Feb 20, 2012 at 1:35 PM, Brett Olsen brett.ol...@gmail.com wrote:
On Sat, Feb 18, 2012 at 8:12 PM, Adam Hughes hugad...@gwmail.gwu.edu wrote:
Hey everyone,
I have timeseries data in which the column label is simply a filename from
which the original data was taken. Here's some sample data:
name1.txt name2.txt name3.txt
32 34 953
32 03 402
I've noticed that the standard genfromtxt() method works great; however, the
names aren't written correctly. That is, if I use the command:
print data['name1.txt']
Nothing happens.
However, when I remove the file extension, Eg:
name1 name2 name3
32 34 953
32 03 402
Then print data['name1'] return (32, 32) as expected. It seems that the
period in the name isn't compatible with the genfromtxt() names attribute.
Is there a workaround, or do I need to restructure my program to get the
extension removed? I'd rather not do this if possible for reasons that
aren't important for the discussion at hand.
It looks like the period is just getting stripped out of the names:
In [1]: import numpy as N
In [2]: N.genfromtxt('sample.txt', names=True)
Out[2]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1txt', 'f8'), ('name2txt', 'f8'), ('name3txt', 'f8')])
Interestingly, this still happens if you supply the names manually:
In [17]: def reader(filename):
: infile = open(filename, 'r')
: names = infile.readline().split()
: data = N.genfromtxt(infile, names=names)
: infile.close()
: return data
:
In [20]: data = reader('sample.txt')
In [21]: data
Out[21]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1txt', 'f8'), ('name2txt', 'f8'), ('name3txt', 'f8')])
What you can do is reset the names after genfromtxt is through with it,
though:
In [34]: def reader(filename):
: infile = open(filename, 'r')
: names = infile.readline().split()
: infile.close()
: data = N.genfromtxt(filename, names=True)
: data.dtype.names = names
: return data
:
In [35]: data = reader('sample.txt')
In [36]: data
Out[36]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1.txt', 'f8'), ('name2.txt', 'f8'), ('name3.txt', 'f8')])
Be warned, I don't know why the period is getting stripped; there may
be a good reason, and adding it in might cause problems.
I think the period is stripped because recarrays also offer attribute
access of names. So you wouldn't be able to do
your_array.sample.txt
All the names get passed through a name validator. IIRC it's something like
from numpy.lib import _iotools
validator = _iotools.NameValidator()
validator.validate('sample1.txt')
validator.validate('a name with spaces')
NameValidator has a good docstring and the gist of this should be in
the genfromtxt docs, if it's not already.
Skipper
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Forbidden charcter in the names argument of genfromtxt?
Thanks for clearing that up.
On Mon, Feb 20, 2012 at 1:58 PM, Skipper Seabold jsseab...@gmail.comwrote:
On Mon, Feb 20, 2012 at 1:35 PM, Brett Olsen brett.ol...@gmail.com
wrote:
On Sat, Feb 18, 2012 at 8:12 PM, Adam Hughes hugad...@gwmail.gwu.edu
wrote:
Hey everyone,
I have timeseries data in which the column label is simply a filename
from
which the original data was taken. Here's some sample data:
name1.txt name2.txt name3.txt
32 34953
32 03402
I've noticed that the standard genfromtxt() method works great;
however, the
names aren't written correctly. That is, if I use the command:
print data['name1.txt']
Nothing happens.
However, when I remove the file extension, Eg:
name1 name2 name3
32 34953
32 03402
Then print data['name1'] return (32, 32) as expected. It seems that the
period in the name isn't compatible with the genfromtxt() names
attribute.
Is there a workaround, or do I need to restructure my program to get the
extension removed? I'd rather not do this if possible for reasons that
aren't important for the discussion at hand.
It looks like the period is just getting stripped out of the names:
In [1]: import numpy as N
In [2]: N.genfromtxt('sample.txt', names=True)
Out[2]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1txt', 'f8'), ('name2txt', 'f8'), ('name3txt',
'f8')])
Interestingly, this still happens if you supply the names manually:
In [17]: def reader(filename):
: infile = open(filename, 'r')
: names = infile.readline().split()
: data = N.genfromtxt(infile, names=names)
: infile.close()
: return data
:
In [20]: data = reader('sample.txt')
In [21]: data
Out[21]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1txt', 'f8'), ('name2txt', 'f8'), ('name3txt',
'f8')])
What you can do is reset the names after genfromtxt is through with it,
though:
In [34]: def reader(filename):
: infile = open(filename, 'r')
: names = infile.readline().split()
: infile.close()
: data = N.genfromtxt(filename, names=True)
: data.dtype.names = names
: return data
:
In [35]: data = reader('sample.txt')
In [36]: data
Out[36]:
array([(32.0, 34.0, 954.0), (32.0, 3.0, 402.0)],
dtype=[('name1.txt', 'f8'), ('name2.txt', 'f8'), ('name3.txt',
'f8')])
Be warned, I don't know why the period is getting stripped; there may
be a good reason, and adding it in might cause problems.
I think the period is stripped because recarrays also offer attribute
access of names. So you wouldn't be able to do
your_array.sample.txt
All the names get passed through a name validator. IIRC it's something like
from numpy.lib import _iotools
validator = _iotools.NameValidator()
validator.validate('sample1.txt')
validator.validate('a name with spaces')
NameValidator has a good docstring and the gist of this should be in
the genfromtxt docs, if it's not already.
Skipper
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On 18/02/12 04:54, Sturla Molden wrote: This is not true. C++ can be much easier, particularly for those who already know Python. The problem: C++ textbooks teach C++ as a subset of C. Writing C in C++ just adds the complexity of C++ on top of C, for no good reason. I can write FORTRAN in any language, it does not mean it is a good idea. We would have to start by teaching people to write good C++. E.g., always use the STL like Python built-in types if possible. Dynamic memory should be std::vector, not new or malloc. Pointers should be replaced with references. We would have to write a C++ programming tutorial that is based on Pyton knowledge instead of C knowledge. Hello Sturla, unrelated to the numpy tewrite debate, can you please suggest some resources you think can be used to learn how to program C++ the proper way? Thank you. Cheers, -- Daniele ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
2012/2/20 Daniele Nicolodi dani...@grinta.net On 18/02/12 04:54, Sturla Molden wrote: This is not true. C++ can be much easier, particularly for those who already know Python. The problem: C++ textbooks teach C++ as a subset of C. Writing C in C++ just adds the complexity of C++ on top of C, for no good reason. I can write FORTRAN in any language, it does not mean it is a good idea. We would have to start by teaching people to write good C++. E.g., always use the STL like Python built-in types if possible. Dynamic memory should be std::vector, not new or malloc. Pointers should be replaced with references. We would have to write a C++ programming tutorial that is based on Pyton knowledge instead of C knowledge. Hello Sturla, unrelated to the numpy tewrite debate, can you please suggest some resources you think can be used to learn how to program C++ the proper way? One of the best books may be Accelerated C++ or the new Stroutrup's book (not the C++ language) Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray and lazy evaluation
On Mon, Feb 20, 2012 at 12:28 PM, Francesc Alted franc...@continuum.iowrote: On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting… Hmm, that would be cute indeed. Do you have an idea on how the code in the with context could be passed to the Python AST compiler (à la numexpr.evaluate(A + B + C))? The biggest problem with the numexpr approach (e.g. evaluate(A + B + C)) whether the programmer has to type the quotes or not, is that the sub-program has to be completely expressed in the sub-language. If I write def f(x): return x[:3] numexpr.evaluate(A + B + f(C)) I would like that to be fast, but it's not obvious at all how that would work. We would be asking numexpr to introspect arbitrary callable python objects, and recompile arbitrary Python code, effectively setting up the expectation in the user's mind that numexpr is re-implementing an entire compiler. That can be fast obviously, but it seems to me to represent significant departure from numpy's focus, which I always thought was the data-container rather than the expression evaluation (though maybe this firestorm of discussion is aimed at changing this?) Theano went with another option which was to replace the A, B, and C variables with objects that have a modified __add__. Theano's back-end can be slow at times and the codebase can feel like a heavy dependency, but my feeling is still that this is a great approach to getting really fast implementations of compound expressions. The context syntax you suggest using is a little ambiguous in that the indented block of a with statement block includes *statements* whereas what you mean to build in the indented block is a *single expression* graph. You could maybe get the right effect with something like A, B, C = np.random.rand(3, 5) expr = np.compound_expression() with np.expression_builder(expr) as foo: arr = A + B + C brr = A + B * C foo.return((arr, brr)) # compute arr and brr as quickly as possible a, b = expr.run() # modify one of the arrays that the expression was compiled to use A[:] += 1 # re-run the compiled expression on the new value a, b = expr.run() - JB -- James Bergstra, Ph.D. Research Scientist Rowland Institute, Harvard University ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray and lazy evaluation
On Mon, Feb 20, 2012 at 1:01 PM, James Bergstra james.bergs...@gmail.comwrote: On Mon, Feb 20, 2012 at 12:28 PM, Francesc Alted franc...@continuum.iowrote: On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting… Hmm, that would be cute indeed. Do you have an idea on how the code in the with context could be passed to the Python AST compiler (à la numexpr.evaluate(A + B + C))? The biggest problem with the numexpr approach (e.g. evaluate(A + B + C)) whether the programmer has to type the quotes or not, is that the sub-program has to be completely expressed in the sub-language. If I write def f(x): return x[:3] numexpr.evaluate(A + B + f(C)) I would like that to be fast, but it's not obvious at all how that would work. We would be asking numexpr to introspect arbitrary callable python objects, and recompile arbitrary Python code, effectively setting up the expectation in the user's mind that numexpr is re-implementing an entire compiler. That can be fast obviously, but it seems to me to represent significant departure from numpy's focus, which I always thought was the data-container rather than the expression evaluation (though maybe this firestorm of discussion is aimed at changing this?) Theano went with another option which was to replace the A, B, and C variables with objects that have a modified __add__. Theano's back-end can be slow at times and the codebase can feel like a heavy dependency, but my feeling is still that this is a great approach to getting really fast implementations of compound expressions. The context syntax you suggest using is a little ambiguous in that the indented block of a with statement block includes *statements* whereas what you mean to build in the indented block is a *single expression* graph. You could maybe get the right effect with something like A, B, C = np.random.rand(3, 5) expr = np.compound_expression() with np.expression_builder(expr) as foo: arr = A + B + C brr = A + B * C foo.return((arr, brr)) # compute arr and brr as quickly as possible a, b = expr.run() # modify one of the arrays that the expression was compiled to use A[:] += 1 # re-run the compiled expression on the new value a, b = expr.run() - JB I should add that the biggest benefit of expressing things as compound expressions in this way is not in saving temporaries (though that is nice) it's being able to express enough computation work at a time that it offsets the time required to ship the arguments off to a GPU for evaluation! This has been a *huge* win reaped by the Theano approach, it works really well. The abstraction boundary offered by this sort of expression graph has been really effective. This speaks even more to the importance of distinguishing between the data container (e.g. numpy, Theano's internal ones, PyOpenCL's one, PyCUDA's one) and the expression compilation and evaluation infrastructures (e.g. Theano, numexpr, cython). The goal should be as much as possible to separate these two so that programs can be expressed in a natural way, and then evaluated using containers that are suited to the program. - JB -- James Bergstra, Ph.D. Research Scientist Rowland Institute, Harvard University ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Francesc Alted writes:
On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
You need at least a slightly different Python API to get anywhere, so
numexpr/Theano is the right place to work on an implementation of this
idea. Of course it would be nice if numexpr/Theano offered something as
convenient as
with lazy:
arr = A + B + C # with all of these NumPy arrays
# compute upon exiting…
Hmm, that would be cute indeed. Do you have an idea on how the code in the
with
context could be passed to the Python AST compiler (à la numexpr.evaluate(A
+ B
+ C))?
Well, I started writing some experiments to almost transparently translate
regular ndarray operations to numexpr strings (or others) using only python
code.
The concept is very simple:
# you only need the first one to start building the AST
a = lazy(np.arange(16))
b = np.arange(16)
res = a + b + 3
print evaluate(res)
# the actual evaluation can be delayed to something like __repr__ or __str__
print repr(res)
print res
# you could also delay evaluation until someone uses res to create a new
array
My target was to use this to also generate optimized GPU kernels in-flight using
pycuda, but I think some other relatively recent project already performed
something similar (w.r.t. generating cuda kernels out of python expressions).
The supporting code for numexpr was something like:
import numexpr
import numpy as np
def build_arg_expr (arg, args):
if isinstance(arg, Expr):
# recursively build the expression
arg_expr, arg_args = arg.build_expr()
args.update(arg_args)
return arg_expr
else:
# unique argument identifier
arg_id = arg_%d % id(arg)
args[arg_id] = arg
return arg_id
# generic expression builder
class Expr:
def evaluate(self):
expr, args = self.build_expr()
return numexpr.evaluate(expr, local_dict = args, global_dict = {})
def __repr__ (self):
return self.evaluate().__repr__()
def __str__ (self):
return self.evaluate().__str__()
def __add__ (self, other):
return ExprAdd(self, other)
# expression builder for adds
class ExprAdd(Expr):
def __init__(self, arg1, arg2):
self.arg1 = arg1
self.arg2 = arg2
def build_expr(self):
args = {}
expr1 = build_arg_expr(self.arg1, args)
expr2 = build_arg_expr(self.arg2, args)
return (+expr1+) + (+expr2+), args
# ndarray-like class to generate expression builders
class LazyNdArray(np.ndarray):
def __add__ (self, other):
return ExprAdd(self, other)
# build a LazyNdArray
def lazy (arg):
return arg.view(LazyNdArray)
# evaluate with numexpr an arbitrary expression builder
def evaluate(arg):
return arg.evaluate()
The thing here is to always return to the user something that looks like an
ndarray.
As you can see the whole thing is not very complex, but some less funny code had
to be written meanwhile for work and I just dropped this :)
Lluis
--
And it's much the same thing with knowledge, for whenever you learn
something new, the whole world becomes that much richer.
-- The Princess of Pure Reason, as told by Norton Juster in The Phantom
Tollbooth
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Charles R Harris wrote: On Fri, Feb 17, 2012 at 12:09 PM, Benjamin Root ben.r...@ou.edu wrote: On Fri, Feb 17, 2012 at 1:00 PM, Christopher Jordan-Squire cjord...@uw.edu wrote: On Fri, Feb 17, 2012 at 10:21 AM, Mark Wiebe mwwi...@gmail.com wrote: On Fri, Feb 17, 2012 at 11:52 AM, Eric Firing efir...@hawaii.edu wrote: On 02/17/2012 05:39 AM, Charles R Harris wrote: On Fri, Feb 17, 2012 at 8:01 AM, David Cournapeau courn...@gmail.com mailto:courn...@gmail.com wrote: Hi Travis, On Thu, Feb 16, 2012 at 10:39 PM, Travis Oliphant tra...@continuum.io mailto:tra...@continuum.io wrote: Mark Wiebe and I have been discussing off and on (as well as talking with Charles) a good way forward to balance two competing desires: * addition of new features that are needed in NumPy * improving the code-base generally and moving towards a more maintainable NumPy I know there are load voices for just focusing on the second of these and avoiding the first until we have finished that. I recognize the need to improve the code base, but I will also be pushing for improvements to the feature-set and user experience in the process. As a result, I am proposing a rough outline for releases over the next year: * NumPy 1.7 to come out as soon as the serious bugs can be eliminated. Bryan, Francesc, Mark, and I are able to help triage some of those. * NumPy 1.8 to come out in July which will have as many ABI-compatible feature enhancements as we can add while improving test coverage and code cleanup. I will post to this list more details of what we plan to address with it later.Included for possible inclusion are: * resolving the NA/missing-data issues * finishing group-by * incorporating the start of label arrays * incorporating a meta-object * a few new dtypes (variable-length string, varialbe-length unicode and an enum type) * adding ufunc support for flexible dtypes and possibly structured arrays * allowing generalized ufuncs to work on more kinds of arrays besides just contiguous * improving the ability for NumPy to receive JIT-generated function pointers for ufuncs and other calculation opportunities * adding filters to Input and Output * simple computed fields for dtypes * accepting a Data-Type specification as a class or JSON file * work towards improving the dtype-addition mechanism * re-factoring of code so that it can compile with a C++ compiler and be minimally dependent on Python data-structures. This is a pretty exciting list of features. What is the rationale for code being compiled as C++ ? IMO, it will be difficult to do so without preventing useful C constructs, and without removing some of the existing features (like our use of C99 complex). The subset that is both C and C++ compatible is quite constraining. I'm in favor of this myself, C++ would allow a lot code cleanup and make it easier to provide an extensible base, I think it would be a natural fit with numpy. Of course, some C++ projects become tangled messes of inheritance, but I'd be very interested in seeing what a good C++ designer like Mark, intimately familiar with the numpy code base, could do. This opportunity might not come by again anytime soon and I think we should grab onto it. The initial step would be a release whose code that would compile in both C/C++, which mostly comes down to removing C++ keywords like 'new'. I did suggest running it by you for build issues, so please raise any you can think of. Note that MatPlotLib is in C++, so I don't think the problems are insurmountable. And choosing a set of compilers to support is something that will need to be done. It's true that matplotlib relies heavily on C++, both via the Agg library and in its own extension code. Personally, I don't like this; I think it raises the barrier to contributing. C++ is an order of magnitude more complicated than C--harder to read, and much harder to write, unless one is a true expert. In mpl it brings reliance on the CXX library, which Mike D. has had to help maintain. And if it does increase compiler specificity, that's bad. This gets to the recruitment issue, which is one of the most important problems I see numpy facing. I personally have contributed a lot of code to NumPy *in spite of* the fact it's in C. NumPy being in C instead of C++ was the biggest negative point when I considered whether
Re: [Numpy-discussion] Proposed Roadmap Overview
Lluís writes: Francesc Alted writes: On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting… Hmm, that would be cute indeed. Do you have an idea on how the code in the with context could be passed to the Python AST compiler (à la numexpr.evaluate(A + B + C))? Well, I started writing some experiments to almost transparently translate regular ndarray operations to numexpr strings (or others) using only python code. [...] My target was to use this to also generate optimized GPU kernels in-flight using pycuda, but I think some other relatively recent project already performed something similar (w.r.t. generating cuda kernels out of python expressions). Aaahhh, I just had a quick look at Theano and it seems it's the project I was referring to. Good job! :) Lluis -- And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer. -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On 20. feb. 2012, at 16:29, Sturla Molden wrote: Den 20.02.2012 08:35, skrev Paul Anton Letnes: In the language wars, I have one question. Why is Fortran not being considered? Fortran already implements many of the features that we want in NumPy: Yes ... but it does not make Fortran a systems programming language. Making NumPy is different from using it. - slicing and similar operations, at least some of the fancy indexing kind - element-wise array operations and function calls - array bounds-checking and other debugging aid (with debugging flags) That is nice for numerical computing, but not really needed to make NumPy. - arrays that mentally map very well onto numpy arrays. To me, this spells +1 to ease of contribution, over some abstract C/C++ template Mentally perhaps, but not binary. NumPy needs uniformly strided memory on the binary level. Fortran just gives this at the mental level. E.g. there is nothing that dictates a Fortran pointer has to be a view, the compiler is free to employ copy-in copy-out. In Fortran, a function call can invalidate a pointer. One would therefore have to store the array in an array of integer*1, and use the intrinsic function transfer() to parse the contents into NumPy dtypes. - in newer standards it has some nontrivial mathematical functions: gamma, bessel, etc. that numpy lacks right now That belongs to SciPy. I don't see exactly why. Why should numpy have exponential but not gamma functions? The division seems kinda arbitrary. Not that I am arguing violently for bessel functions in numpy. - compilers that are good at optimizing for floating-point performance, because that's what Fortran is all about Insanely good, but not when we start to do the (binary, not mentally) strided access that NumPy needs. (Not that C compilers would be any better.) - not Fortran as such, but BLAS and LAPACK are easily accessed by Fortran - possibly other numerical libraries that can be helpful - Fortran has, in its newer standards, thought of C interoperability. We could still keep bits of the code in C (or even C++?) if we'd like to, or perhaps f2py/Cython could do the wrapping. Not f2py, as it depends on NumPy. - some programmers know Fortran better than C++. Fortran is at least used by many science guys, like me. That is a valid arguments. Fortran is also much easier to read and debug. Sturla Thanks for an excellent answer, Sturla - very informative indeed. Paul. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray and lazy evaluation
James Bergstra writes: [...] I should add that the biggest benefit of expressing things as compound expressions in this way is not in saving temporaries (though that is nice) it's being able to express enough computation work at a time that it offsets the time required to ship the arguments off to a GPU for evaluation! Right, that's exacly what you need for an external computation to pay off. Just out of curiosity (feel free to respond with a RTFM or a RTFP :)), do you support any of these? (sorry for the made-up names) * automatic transfer double-buffering * automatic problem partitioning into domains (e.g., multiple GPUs; even better if also supports nodes - MPI -) * point-specific computations (e.g., code dependant on the thread id, although this can also be expressed in other ways, like index ranges) * point-relative computations (the most common would be a stencil) If you have all of them, then I'd say the project has a huge potential for total world dominance :) Lluis -- And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer. -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Looks like Dag forked the discussion of lazy evaluation to a new thread ([Numpy-discussion] ndarray and lazy evaluation). There are actually several projects inspired by this sort of design: off the top of my head I can think of Theano, copperhead, numexpr, arguably sympy, and some non-public code by Nicolas Pinto. So I think the strengths of the approach in principle are established... the big question is how to make this approach easy to use in all the settings where it could be useful. I don't think any of these projects has gotten that totally right. -JB On Mon, Feb 20, 2012 at 2:41 PM, Lluís xscr...@gmx.net wrote: Lluís writes: Francesc Alted writes: On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote: You need at least a slightly different Python API to get anywhere, so numexpr/Theano is the right place to work on an implementation of this idea. Of course it would be nice if numexpr/Theano offered something as convenient as with lazy: arr = A + B + C # with all of these NumPy arrays # compute upon exiting… Hmm, that would be cute indeed. Do you have an idea on how the code in the with context could be passed to the Python AST compiler (à la numexpr.evaluate(A + B + C))? Well, I started writing some experiments to almost transparently translate regular ndarray operations to numexpr strings (or others) using only python code. [...] My target was to use this to also generate optimized GPU kernels in-flight using pycuda, but I think some other relatively recent project already performed something similar (w.r.t. generating cuda kernels out of python expressions). Aaahhh, I just had a quick look at Theano and it seems it's the project I was referring to. Good job! :) Lluis -- And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer. -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- http://www-etud.iro.umontreal.ca/~bergstrj ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Den 20.02.2012 20:14, skrev Daniele Nicolodi: Hello Sturla, unrelated to the numpy tewrite debate, can you please suggest some resources you think can be used to learn how to program C++ the proper way? Thank you. Cheers, This is totally OT on this list, however ... Scott Meyer's books have been mentioned. Also look at some litterature on the STL (e.g. Josuittis). Getting the Boost library is essential as well. The Qt library have many examples of beautiful C++. But the most important part, in my opinion, is to put the C with classes mentality away. Look at it as compiled Python or Java. The STL (the standard C++ library) has classes that do the same as the types we use in Python --- there are parallels to tuple, dict, set, list, deque, etc. The STL is actually richer than Python. Just use them the way we use Python. With C++11 (the latest standard), even for loops can be like Python. There are lamdas and closures, to be used as in Python, and there is an 'auto' keyword for type inference; you don't have to declare the type of a variable, the compiler will figure it out. Don't use new[] just because you can, when there is std::vector that behaves lika Python list. If you need to allocate a resource, wrap it in a class. Allocate from the contructor and deallocate from the destructor. That way an exception cannot cause a resource leak, and the clean-up code will be called automatically when the object fall of the stack. If you need to control the lifetime of an object, make an inner block with curly brackets, and declare it on top of the block. Don't call new and delete to control where you want it to be allocated and deallocated. Nothing goes on the heap unless STL puts it there. Always put objects on the stack, never allocate to a pointer with new. Always use references, and forget about pointers. This has to do with putting the C with classes mentality away. Always implement a copy constructor so the classes work with the STL. std:: vectordouble x(n); // ok void foobar(std:: vectordouble x); // ok double* x = new double [n]; // bad std:: vectordouble *x = new std:: vectordouble (n); // bad void foobar(std:: vectordouble* x); // bad If you get any textbook on Windows programming from Microsoft Press, you have an excellent resource on what not to do. Verbose functions and field names, Hungarian notation, factories instead of constructors, etc. If you find yourself using macros or template magic to avoid the overhead of a virtual function (MFC, ATL, wxWidgets, FOX), for the expense of readability, you are probably doing something you shouldn't. COM is probably the worst example I know of, just compare the beautiful OpenGL to Direct3D. VTK is another example of what I consider ugly C++. But that's just my opinion. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray and lazy evaluation
On Mon, Feb 20, 2012 at 2:57 PM, Lluís xscr...@gmx.net wrote: James Bergstra writes: [...] I should add that the biggest benefit of expressing things as compound expressions in this way is not in saving temporaries (though that is nice) it's being able to express enough computation work at a time that it offsets the time required to ship the arguments off to a GPU for evaluation! Right, that's exacly what you need for an external computation to pay off. Just out of curiosity (feel free to respond with a RTFM or a RTFP :)), do you support any of these? (sorry for the made-up names) * automatic transfer double-buffering Not currently, but it would be quite straightforward to do it. Email theano-dev and ask how if you really want to know. * automatic problem partitioning into domains (e.g., multiple GPUs; even better if also supports nodes - MPI -) Not currently, and it would be hard. * point-specific computations (e.g., code dependant on the thread id, although this can also be expressed in other ways, like index ranges) No. * point-relative computations (the most common would be a stencil) No, but I think theano provides a decent expression language to tackle this. The Composite element-wise code generator is an example of how I would think about this. It provides point-relative computations across several arguments. You might want something different that applies a stencil computation across one or several arguments... the scan operator was another foray into this territory, and it got tricky when the stencil operation could have side-effects (like random number generation) and could define it's own input domain (stencil shape), but the result is quite powerful. -- http://www-etud.iro.umontreal.ca/~bergstrj ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Mon, Feb 20, 2012 at 19:55, Paul Anton Letnes paul.anton.let...@gmail.com wrote: On 20. feb. 2012, at 16:29, Sturla Molden wrote: - in newer standards it has some nontrivial mathematical functions: gamma, bessel, etc. that numpy lacks right now That belongs to SciPy. I don't see exactly why. Why should numpy have exponential but not gamma functions? The division seems kinda arbitrary. Not that I am arguing violently for bessel functions in numpy. The semi-arbitrary dividing line that we have settled on is C99. If a special function is in the C99 standard, we'll accept an implementation for it in numpy. Part (well, most) of the rationale is just to have a clear dividing line even if it's fairly arbitrary. The other part is that if a decidedly non-mathematically-focused standard like C99 includes a special function in its standard library, then odds are good that it's something that is widely used enough as a building block for other things. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Interesting you bring this up. I actually have a working prototype of using
Python to emit LLVM. I will be showing it at the HPC tutorial that I am
giving at PyCon.I will be making this available after PyCon to a wider
audience as open source.
It uses llvm-py (modified to work with LLVM 3.0) and code I wrote to do the
translation from Python byte-code to LLVM. This LLVM can then be JITed. I
have several applications that I would like to use this for. It would be
possible to write more of NumPy using this approach. Initially, it makes
it *very* easy to create a machine-code ufunc from Python code. There are
other use-cases of having loops written in Python and plugged in to a
calculation, filtering, or indexing framework that this system will be useful
for.
There is still a need for a core data-type object, a core array object, and a
core calculation object. Maybe some-day these cores can be shrunk to a
smaller subset and more of something along the lines of LLVM generation from
Python can be used. But, there is a lot of work to do before that is
possible.But, a lot of the currently pre-compiled loops can be done on the
fly instead using this approach.There are several things I'm working on in
that direction.
This is not PyPy. It certainly uses the same ideas that they are using, but
instead it fits into the CPython run-time and doesn't require changing the
whole ecosystem. If you are interested in this work let me know. I think
I'm going to call the project numpy-llvm, or fast-py, or something like that.
It is available on github and will be open source (but it's still under active
development).
Here is an example of the code to create a ufunc using the system (this is like
vectorize, but it creates machine code and by-passes the interpreter and so is
100x faster).
from math import sin, pi
def sinc(x):
if x==0:
return 1.0
else:
return sin(x*pi)/(pi*x)
from translate import Translate
t = Translate(sinc)
t.translate()
print t.mod
res = t.make_ufunc('sinc')
-Travis
On Feb 20, 2012, at 10:55 AM, Sturla Molden wrote:
Den 20.02.2012 17:42, skrev Sturla Molden:
There are still other options than C or C++ that are worth considering.
One would be to write NumPy in Python. E.g. we could use LLVM as a
JIT-compiler and produce the performance critical code we need on the fly.
LLVM and its C/C++ frontend Clang are BSD licenced. It compiles faster
than GCC and often produces better machine code. They can therefore be
used inside an array library. It would give a faster NumPy, and we could
keep most of it in Python.
Sturla
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] is there an efficient way to get a random set of subsets/combinations?
Hi to all Numeric Python experts, could not think of a mailing list with better fit to my question which might have an obvious answer: straightforward (naive) Python code to answer my question would be something like import random, itertools n,p,k=100,50,10 # don't try to run with this numbers! ;) print random.sample(list(itertools.combinations(range(n), p)), k) so the goal is to get k (non-repeating) p-subsets of n, where n and p prohibitively large to first populate the full set of combinations. Thank you in advance ;-) -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is there an efficient way to get a random set of subsets/combinations?
If you're using numpy 2.0 (the development branch), the function numpy.random.choice might do what you're looking for. -Chris On Mon, Feb 20, 2012 at 8:35 PM, Yaroslav Halchenko li...@onerussian.com wrote: Hi to all Numeric Python experts, could not think of a mailing list with better fit to my question which might have an obvious answer: straightforward (naive) Python code to answer my question would be something like import random, itertools n,p,k=100,50,10 # don't try to run with this numbers! ;) print random.sample(list(itertools.combinations(range(n), p)), k) so the goal is to get k (non-repeating) p-subsets of n, where n and p prohibitively large to first populate the full set of combinations. Thank you in advance ;-) -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] is there an efficient way to get a random set of subsets/combinations?
Hi Slava, Since your k is only 10, here is a quickie: import numpy as np arr = np.arange(n) for i in range(k): np.random.shuffle(arr) print np.sort(arr[:p]) If your ever get non-unique entries in a set of k=10 for your n and p, consider yourself lucky:) Val On Mon, Feb 20, 2012 at 10:35 PM, Yaroslav Halchenko li...@onerussian.comwrote: Hi to all Numeric Python experts, could not think of a mailing list with better fit to my question which might have an obvious answer: straightforward (naive) Python code to answer my question would be something like import random, itertools n,p,k=100,50,10 # don't try to run with this numbers! ;) print random.sample(list(itertools.combinations(range(n), p)), k) so the goal is to get k (non-repeating) p-subsets of n, where n and p prohibitively large to first populate the full set of combinations. Thank you in advance ;-) -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
