Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
Excerpts from Nathaniel Smith's message of Tue Feb 28 17:22:16 -0500 2012: Even for binary, there are pathological cases, e.g. 1) reading a random subset of nearly all rows. 2) reading a single column when rows are small. In case 2 you will only go this route in the first place if you need to save memory. The user should be aware of these issues. FWIW, this route actually doesn't save any memory as compared to np.memmap. Actually, for numpy.memmap you will read the whole file if you try to grab a single column and read a large fraction of the rows. Here is an example that will end up pulling the entire file into memory mm=numpy.memmap(fname, dtype=dtype) rows=numpy.arange(mm.size) x=mm['x'][rows] I just tested this on a 3G binary file and I'm sitting at 3G memory usage. I believe this is because numpy.memmap only understands rows. I don't fully understand the reason for that, but I suspect it is related to the fact that the ndarray really only has a concept of itemsize, and the fields are really just a reinterpretation of those bytes. It may be that one could tweak the ndarray code to get around this. But I would appreciate enlightenment on this subject. This fact was the original motivator for writing my code; the text reading ability came later. Cool. I'm just a little concerned that, since we seem to have like... 5 different implementations of this stuff all being worked on at the same time, we need to get some consensus on which features actually matter, so they can be melded together into the Single Best File Reader Evar. An interface where indexing and file-reading are combined is significantly more complicated than one where the core file-reading inner-loop can ignore indexing. So far I'm not sure why this complexity would be worthwhile, so that's what I'm trying to understand. I think I've addressed the reason why the low level C code was written. And I think a unified, high level interface to binary and text files, which the Recfile class provides, is worthwhile. Can you please say more about ...one where the core file-reading inner-loop can ignore indexing? I didn't catch the meaning. -e Cheers, -- Nathaniel Also, for some crazy ascii files we may want to revert to pure python anyway, but I think these should be special cases that can be flagged at runtime through keyword arguments to the python functions. BTW, did you mean to go off-list? cheers, -e -- Erin Scott Sheldon Brookhaven National Laboratory -- Erin Scott Sheldon Brookhaven National Laboratory ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
Excerpts from Erin Sheldon's message of Wed Feb 29 10:11:51 -0500 2012: Actually, for numpy.memmap you will read the whole file if you try to grab a single column and read a large fraction of the rows. Here is an That should have been: ...read *all* the rows. -e -- Erin Scott Sheldon Brookhaven National Laboratory ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] [Numpy] quadruple precision
I am completely new to Numpy and I know only the basics of Python, to this point I was using Fortran 03/08 to write numerical code. However, I am starting a new large project of mine and I am looking forward to using Python to call some low level Fortran code responsible for most of the intensive number crunching. In this context I stumbled into f2py and it looks just like what I need, but before I start writing an app in mixture of Python and Fortran I have a question about numerical precision of variables used in numpy and f2py. Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I have investigated the float128 data type, but it seems to work as binary64 or binary80 depending on the architecture. If there is currently no way to interact with binary128, how hard would it be to patch the sources of numpy to add such data type? I am interested only in basic stuff, comparable in functionality to libmath. As said before, I have little knowledge of Python, Numpy and f2py, I am however, interested in investing some time in learing it and implementing the mentioned features, but only if there is any hope of succeeding. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
On Wed, Feb 29, 2012 at 15:11, Erin Sheldon erin.shel...@gmail.com wrote: Excerpts from Nathaniel Smith's message of Tue Feb 28 17:22:16 -0500 2012: Even for binary, there are pathological cases, e.g. 1) reading a random subset of nearly all rows. 2) reading a single column when rows are small. In case 2 you will only go this route in the first place if you need to save memory. The user should be aware of these issues. FWIW, this route actually doesn't save any memory as compared to np.memmap. Actually, for numpy.memmap you will read the whole file if you try to grab a single column and read a large fraction of the rows. Here is an example that will end up pulling the entire file into memory mm=numpy.memmap(fname, dtype=dtype) rows=numpy.arange(mm.size) x=mm['x'][rows] I just tested this on a 3G binary file and I'm sitting at 3G memory usage. I believe this is because numpy.memmap only understands rows. I don't fully understand the reason for that, but I suspect it is related to the fact that the ndarray really only has a concept of itemsize, and the fields are really just a reinterpretation of those bytes. It may be that one could tweak the ndarray code to get around this. But I would appreciate enlightenment on this subject. Each record (I would avoid the word row in this context) is contiguous in memory whether that memory is mapped to disk or not. Additionally, the way that virtual memory (i.e. mapped memory) works is that when you request the data at a given virtual address, the OS will go look up the page it resides in (typically 4-8k in size) and pull the whole page into main memory. Since you are requesting most of the records, you are probably pulling all of the file into main memory. Memory mapping works best when you pull out contiguous chunks at a time rather than pulling out stripes. numpy structured arrays do not rearrange your data to put all of the 'x' data contiguous with each other. You can arrange that yourself, if you like (use a structured scalar with a dtype such that each field is an array of the appropriate length and dtype). Then pulling out all of the 'x' field values will only touch a smaller fraction of the file. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
On Tue, Feb 28, 2012 at 11:28 PM, Mark Wiebe mwwi...@gmail.com wrote: The development approach I really like is to start with a relatively rough NEP, then cycle through feedback, updating the NEP, and implementation. Organizing ones thoughts to describe them in a design document can often clarify things that are confusing when just looking at code. Feedback from the community, both developers and users, can help expose where your assumptions are and often lead to insights from subjects you didn't even know about. Implementation puts those ideas through the a cold, hard, reality check, and can provide a hands-on experience for later rounds of feedback. This iterative process is most important to emphasize, the design document and the code must both evolve together. Stamping a NEP as final before getting into code is just as bad as jumping into code without writing a preliminary design. Certainly! We're in complete agreement here. I didn't mean to suggest (though perhaps I phrased it poorly) that the nep discussion and implementation phases should be fully disjoint, since I do believe that implementation and discussion can and should inform each other. Github actually has a bug that the RST table of contents is stripped, and this makes reading longer NEPS right in the repository uncomfortable. Maybe alternatives to a git repository for NEPs should be considered. I reported the bug to github, but they told me that was just how they did things. That's easy to solve, and can be done with a minimum of work in a way that will make the nep-handling process far eaiser: - split the neps into their own repo, and make that a repo targeted for building a website, like we do with the ipython docs for example. - have a 'nep repo manager' who merges PRs from nep authors quickly. In practice, nep authors could even be given write access to the repo while they work on their own nep, I think we can trust people not to mess around outside their directory. - the nep repo is source-only, and we have a nep-web repo where the *built* neps are displayed using the gh-pages mechanism. With this, we achieve something like what python uses, with a separate and nicely formatted web version of the neps for easy reading, but in addition with the fluidity of the github workflow for source management. We already have all the pieces for this, so it would be a very easy job for someone to make it happen (~2 hours at most, would be my guess). Cheers, f ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
Thanks to your question, I discovered that there is a float128 dtype in numpy In[5]: np.__version__ Out[5]: '1.6.1' In[6]: np.float128? Type: type Base Class: type 'type' String Form:type 'numpy.float128' Namespace: Interactive File: /Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/numpy/__init__.py Docstring: 128-bit floating-point number. Character code: 'g'. C long float compatible. Based on some reported issues, it seems like there are issues though with this and its mapping to python long integer... http://mail.scipy.org/pipermail/numpy-discussion/2011-October/058784.html HTH, Jonathan On Wed, Feb 29, 2012 at 9:22 AM, Paweł Biernat pw...@wp.pl wrote: I am completely new to Numpy and I know only the basics of Python, to this point I was using Fortran 03/08 to write numerical code. However, I am starting a new large project of mine and I am looking forward to using Python to call some low level Fortran code responsible for most of the intensive number crunching. In this context I stumbled into f2py and it looks just like what I need, but before I start writing an app in mixture of Python and Fortran I have a question about numerical precision of variables used in numpy and f2py. Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I have investigated the float128 data type, but it seems to work as binary64 or binary80 depending on the architecture. If there is currently no way to interact with binary128, how hard would it be to patch the sources of numpy to add such data type? I am interested only in basic stuff, comparable in functionality to libmath. As said before, I have little knowledge of Python, Numpy and f2py, I am however, interested in investing some time in learing it and implementing the mentioned features, but only if there is any hope of succeeding. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Jonathan Rocher, PhD Scientific software developer Enthought, Inc. jroc...@enthought.com 1-512-536-1057 http://www.enthought.com ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
Hi, On Wed, Feb 29, 2012 at 12:13 PM, Jonathan Rocher jroc...@enthought.com wrote: Thanks to your question, I discovered that there is a float128 dtype in numpy In[5]: np.__version__ Out[5]: '1.6.1' In[6]: np.float128? Type: type Base Class: type 'type' String Form:type 'numpy.float128' Namespace: Interactive File: /Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/numpy/__init__.py Docstring: 128-bit floating-point number. Character code: 'g'. C long float compatible. Right - but remember that numpy float128 is different on different platforms. In particular, float128 is any C longdouble type that needs 128 bits of memory, regardless of precision or implementation. See [1] for background on C longdouble type. The numpy platforms I know about are: Intel : 80 bit float padded to 128 bits [2] PPC : pair of float64 values [3] Debian IBM s390 : real quadruple precision [4] [5] I see that some Sun machines implement real quadruple precision in software but I haven't run numpy on a Sun machine [6] [1] http://en.wikipedia.org/wiki/Long_double [2] http://en.wikipedia.org/wiki/Extended_precision#x86_Architecture_Extended_Precision_Format [3] http://en.wikipedia.org/wiki/Double-double_%28arithmetic%29#Double-double_arithmetic [4] http://en.wikipedia.org/wiki/Double-double_%28arithmetic%29#IEEE_754_quadruple-precision_binary_floating-point_format:_binary128 [5] https://github.com/nipy/nibabel/issues/76 [6] http://en.wikipedia.org/wiki/Double-double_%28arithmetic%29#Implementations Based on some reported issues, it seems like there are issues though with this and its mapping to python long integer... http://mail.scipy.org/pipermail/numpy-discussion/2011-October/058784.html I tried to summarize the problems I knew about here: http://mail.scipy.org/pipermail/numpy-discussion/2011-November/059087.html There are some routines to deal with some of the problems here: https://github.com/nipy/nibabel/blob/master/nibabel/casting.py After spending some time with the various longdoubles in numpy, I have learned to stare at my code for a long time considering how it might run into the various problems above. Best, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
On Wed, Feb 29, 2012 at 3:11 PM, Erin Sheldon erin.shel...@gmail.com wrote: Excerpts from Nathaniel Smith's message of Tue Feb 28 17:22:16 -0500 2012: Even for binary, there are pathological cases, e.g. 1) reading a random subset of nearly all rows. 2) reading a single column when rows are small. In case 2 you will only go this route in the first place if you need to save memory. The user should be aware of these issues. FWIW, this route actually doesn't save any memory as compared to np.memmap. Actually, for numpy.memmap you will read the whole file if you try to grab a single column and read a large fraction of the rows. Here is an example that will end up pulling the entire file into memory mm=numpy.memmap(fname, dtype=dtype) rows=numpy.arange(mm.size) x=mm['x'][rows] I just tested this on a 3G binary file and I'm sitting at 3G memory usage. I believe this is because numpy.memmap only understands rows. I don't fully understand the reason for that, but I suspect it is related to the fact that the ndarray really only has a concept of itemsize, and the fields are really just a reinterpretation of those bytes. It may be that one could tweak the ndarray code to get around this. But I would appreciate enlightenment on this subject. Ahh, that makes sense. But, the tool you are using to measure memory usage is misleading you -- you haven't mentioned what platform you're on, but AFAICT none of them have very good tools for describing memory usage when mmap is in use. (There isn't a very good way to handle it.) What's happening is this: numpy read out just that column from the mmap'ed memory region. The OS saw this and decided to read the entire file, for reasons discussed previously. Then, since it had read the entire file, it decided to keep it around in memory for now, just in case some program wanted it again in the near future. Now, if you instead fetched just those bytes from the file using seek+read or whatever, the OS would treat that request in the exact same way: it'd still read the entire file, and it would still keep the whole thing around in memory. On Linux, you could test this by dropping caches (echo 1 /proc/sys/vm/drop_caches), checking how much memory is listed as free in top, and then using your code to read the same file -- you'll see that the 'free' memory drops by 3 gigabytes, and the 'buffers' or 'cached' numbers will grow by 3 gigabytes. [Note: if you try this experiment, make sure that you don't have the same file opened with np.memmap -- for some reason Linux seems to ignore the request to drop_caches for files that are mmap'ed.] The difference between mmap and reading is that in the former case, then this cache memory will be counted against your process's resident set size. The same memory is used either way -- it's just that it gets reported differently by your tool. And in fact, this memory is not really used at all, in the way we usually mean that term -- it's just a cache that the OS keeps, and it will immediately throw it away if there's a better use for that memory. The only reason it's loading the whole 3 gigabytes into memory in the first place is that you have 3 gigabytes of memory to spare. You might even be able to tell the OS that you *won't* be reading that file again, so there's no point in keeping it all cached -- on Unix this is done via the madavise() or posix_fadvise() syscalls. (No guarantee the OS will actually listen, though.) This fact was the original motivator for writing my code; the text reading ability came later. Cool. I'm just a little concerned that, since we seem to have like... 5 different implementations of this stuff all being worked on at the same time, we need to get some consensus on which features actually matter, so they can be melded together into the Single Best File Reader Evar. An interface where indexing and file-reading are combined is significantly more complicated than one where the core file-reading inner-loop can ignore indexing. So far I'm not sure why this complexity would be worthwhile, so that's what I'm trying to understand. I think I've addressed the reason why the low level C code was written. And I think a unified, high level interface to binary and text files, which the Recfile class provides, is worthwhile. Can you please say more about ...one where the core file-reading inner-loop can ignore indexing? I didn't catch the meaning. Sure, sorry. What I mean is just, it's easier to write code that only knows how to do a dumb sequential read, and doesn't know how to seek to particular places and pick out just the fields that are being requested. And it's easier to maintain, and optimize, and document, and add features, and so forth. (And we can still have a high-level interface on top of it, if that's useful.) So I'm trying to understand if there's really a compelling advantage that we get by build seeking smarts into our low-level C code, that we can't get
Re: [Numpy-discussion] Proposed Roadmap Overview
Hi, On Wed, Feb 29, 2012 at 1:46 AM, Travis Oliphant tra...@continuum.io wrote: We already use the NEP process for such decisions. This discussion came from simply from the *idea* of writing such a NEP. Nothing has been decided. Only opinions have been shared that might influence the NEP. This is all pretty premature, though --- migration to C++ features on a trial branch is some months away were it to happen. Fernando can correct me if I'm wrong, but I think he was asking a governance question. That is: would you (as BDF$N) consider the following guideline: As a condition for accepting significant changes to Numpy, for each significant change, there will be a NEP. The NEP shall follow the same model as the Python PEPs - that is - there will be a summary of the changes, the issues arising, the for / against opinions and alternatives offered. There will usually be a draft implementation. The NEP will contain the resolution of the discussion as it relates to the code For example, the masked array NEP, although very substantial, contains little discussion of the controversy arising, or the intended resolution of the controversy: https://github.com/numpy/numpy/blob/3f685a1a990f7b6e5149c80b52436fb4207e49f5/doc/neps/missing-data.rst I mean, although it is useful, it is not in the form of a PEP, as Fernando has described it. Would you accept extending the guidelines to the NEP format? Best, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
Excerpts from Nathaniel Smith's message of Wed Feb 29 13:17:53 -0500 2012: On Wed, Feb 29, 2012 at 3:11 PM, Erin Sheldon erin.shel...@gmail.com wrote: Excerpts from Nathaniel Smith's message of Tue Feb 28 17:22:16 -0500 2012: Even for binary, there are pathological cases, e.g. 1) reading a random subset of nearly all rows. 2) reading a single column when rows are small. In case 2 you will only go this route in the first place if you need to save memory. The user should be aware of these issues. FWIW, this route actually doesn't save any memory as compared to np.memmap. Actually, for numpy.memmap you will read the whole file if you try to grab a single column and read a large fraction of the rows. Here is an example that will end up pulling the entire file into memory mm=numpy.memmap(fname, dtype=dtype) rows=numpy.arange(mm.size) x=mm['x'][rows] I just tested this on a 3G binary file and I'm sitting at 3G memory usage. I believe this is because numpy.memmap only understands rows. I don't fully understand the reason for that, but I suspect it is related to the fact that the ndarray really only has a concept of itemsize, and the fields are really just a reinterpretation of those bytes. It may be that one could tweak the ndarray code to get around this. But I would appreciate enlightenment on this subject. Ahh, that makes sense. But, the tool you are using to measure memory usage is misleading you -- you haven't mentioned what platform you're on, but AFAICT none of them have very good tools for describing memory usage when mmap is in use. (There isn't a very good way to handle it.) What's happening is this: numpy read out just that column from the mmap'ed memory region. The OS saw this and decided to read the entire file, for reasons discussed previously. Then, since it had read the entire file, it decided to keep it around in memory for now, just in case some program wanted it again in the near future. Now, if you instead fetched just those bytes from the file using seek+read or whatever, the OS would treat that request in the exact same way: it'd still read the entire file, and it would still keep the whole thing around in memory. On Linux, you could test this by dropping caches (echo 1 /proc/sys/vm/drop_caches), checking how much memory is listed as free in top, and then using your code to read the same file -- you'll see that the 'free' memory drops by 3 gigabytes, and the 'buffers' or 'cached' numbers will grow by 3 gigabytes. [Note: if you try this experiment, make sure that you don't have the same file opened with np.memmap -- for some reason Linux seems to ignore the request to drop_caches for files that are mmap'ed.] The difference between mmap and reading is that in the former case, then this cache memory will be counted against your process's resident set size. The same memory is used either way -- it's just that it gets reported differently by your tool. And in fact, this memory is not really used at all, in the way we usually mean that term -- it's just a cache that the OS keeps, and it will immediately throw it away if there's a better use for that memory. The only reason it's loading the whole 3 gigabytes into memory in the first place is that you have 3 gigabytes of memory to spare. You might even be able to tell the OS that you *won't* be reading that file again, so there's no point in keeping it all cached -- on Unix this is done via the madavise() or posix_fadvise() syscalls. (No guarantee the OS will actually listen, though.) This is interesting, and on my machine I think I've verified that what you say is actually true. This all makes theoretical sense, but goes against some experiments I and my colleagues have done. For example, a colleague of mine was able to read a couple of large files in using my code but not using memmap. The combined files were greater than memory size. With memmap the code started swapping. This was on 32-bit OSX. But as I said, I just tested this on my linux box and it works fine with numpy.memmap. I don't have an OSX box to test this. So if what you say holds up on non-linux systems, it is in fact an indicator that the section of my code dealing with binary could be dropped; that bit was trivial anyway. -e -- Erin Scott Sheldon Brookhaven National Laboratory ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
Charles R Harris wrote: On Tue, Feb 28, 2012 at 12:05 PM, John Hunter jdh2...@gmail.com wrote: On Sat, Feb 18, 2012 at 5:09 PM, David Cournapeau courn...@gmail.comwrote: There are better languages than C++ that has most of the technical benefits stated in this discussion (rust and D being the most obvious ones), but whose usage is unrealistic today for various reasons: knowledge, availability on esoteric platforms, etc… A new language is completely ridiculous. I just saw this for the first time today: Linus Torvalds on C++ ( http://harmful.cat-v.org/software/c++/linus). The post is from 2007 so many of you may have seen it, but I thought it was entertainng enough and on-topic enough with this thread that I'd share it in case you haven't. The point he makes: In other words, the only way to do good, efficient, and system-level and portable C++ ends up to limit yourself to all the things that are basically available in C was interesting to me because the best C++ library I have ever worked with (agg) imports *nothing* except standard C libs (no standard template library). In fact, the only includes external to external to itself are math.h, stdlib.h, stdio.h, and string.h. To shoehorn Jamie Zawinski's famous regex quote ( http://regex.info/blog/2006-09-15/247). Some people, when confronted with a problem, think “I know, I'll use boost.” Now they have two problems. Here is the Linus post: From: Linus Torvalds torvalds at linux-foundation.org Subject: Re: [RFC] Convert builin-mailinfo.c to use The Better String Library. Newsgroups: gmane.comp.version-control.git Date: 2007-09-06 17:50:28 GMT (2 years, 14 weeks, 16 hours and 36 minutes ago) On Wed, 5 Sep 2007, Dmitry Kakurin wrote: When I first looked at Git source code two things struck me as odd: 1. Pure C as opposed to C++. No idea why. Please don't talk about portability, it's BS. *YOU* are full of bullshit. C++ is a horrible language. It's made more horrible by the fact that a lot of substandard programmers use it, to the point where it's much much easier to generate total and utter crap with it. Quite frankly, even if the choice of C were to do *nothing* but keep the C++ programmers out, that in itself would be a huge reason to use C. In other words: the choice of C is the only sane choice. I know Miles Bader jokingly said to piss you off, but it's actually true. I've come to the conclusion that any programmer that would prefer the project to be in C++ over C is likely a programmer that I really *would* prefer to piss off, so that he doesn't come and screw up any project I'm involved with. C++ leads to really really bad design choices. You invariably start using the nice library features of the language like STL and Boost and other total and utter crap, that may help you program, but causes: - infinite amounts of pain when they don't work (and anybody who tells me that STL and especially Boost are stable and portable is just so full of BS that it's not even funny) - inefficient abstracted programming models where two years down the road you notice that some abstraction wasn't very efficient, but now all your code depends on all the nice object models around it, and you cannot fix it without rewriting your app. In other words, the only way to do good, efficient, and system-level and portable C++ ends up to limit yourself to all the things that are basically available in C. And limiting your project to C means that people don't screw that up, and also means that you get a lot of programmers that do actually understand low-level issues and don't screw things up with any idiotic object model crap. So I'm sorry, but for something like git, where efficiency was a primary objective, the advantages of C++ is just a huge mistake. The fact that we also piss off people who cannot see that is just a big additional advantage. If you want a VCS that is written in C++, go play with Monotone. Really. They use a real database. They use nice object-oriented libraries. They use nice C++ abstractions. And quite frankly, as a result of all these design decisions that sound so appealing to some CS people, the end result is a horrible and unmaintainable mess. But I'm sure you'd like it more than git. Yeah, Linus doesn't like C++. No doubt that is in part because of the attempt to rewrite Linux in C++ back in the early 90's and the resulting compiler and portability problems. Linus also writes C like it was his native tongue, he likes to work close to the metal, and he'd probably prefer it over Python for most problems ;) Things have improved in the compiler department, and I think C++ really wasn't much of an improvement over C until templates and the STL came along. The boost smart pointers are also really nice. OTOH, it is really easy to write awful C++ because of the way inheritance and the other features were over-hyped and the
Re: [Numpy-discussion] Proposed Roadmap Overview
On Wed, Feb 29, 2012 at 1:20 PM, Neal Becker ndbeck...@gmail.com wrote: Much of Linus's complaints have to do with the use of c++ in the _kernel_. These objections are quite different for an _application_. For example, there are issues with the need for support libraries for exception handling. Not an issue for an application. Actually, the thread was on the git mailing list, and many of his complaints were addressing the appropriateness of C++ for git development. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers
On Wed, Feb 29, 2012 at 7:57 PM, Erin Sheldon erin.shel...@gmail.comwrote: Excerpts from Nathaniel Smith's message of Wed Feb 29 13:17:53 -0500 2012: On Wed, Feb 29, 2012 at 3:11 PM, Erin Sheldon erin.shel...@gmail.com wrote: Excerpts from Nathaniel Smith's message of Tue Feb 28 17:22:16 -0500 2012: Even for binary, there are pathological cases, e.g. 1) reading a random subset of nearly all rows. 2) reading a single column when rows are small. In case 2 you will only go this route in the first place if you need to save memory. The user should be aware of these issues. FWIW, this route actually doesn't save any memory as compared to np.memmap. Actually, for numpy.memmap you will read the whole file if you try to grab a single column and read a large fraction of the rows. Here is an example that will end up pulling the entire file into memory mm=numpy.memmap(fname, dtype=dtype) rows=numpy.arange(mm.size) x=mm['x'][rows] I just tested this on a 3G binary file and I'm sitting at 3G memory usage. I believe this is because numpy.memmap only understands rows. I don't fully understand the reason for that, but I suspect it is related to the fact that the ndarray really only has a concept of itemsize, and the fields are really just a reinterpretation of those bytes. It may be that one could tweak the ndarray code to get around this. But I would appreciate enlightenment on this subject. Ahh, that makes sense. But, the tool you are using to measure memory usage is misleading you -- you haven't mentioned what platform you're on, but AFAICT none of them have very good tools for describing memory usage when mmap is in use. (There isn't a very good way to handle it.) What's happening is this: numpy read out just that column from the mmap'ed memory region. The OS saw this and decided to read the entire file, for reasons discussed previously. Then, since it had read the entire file, it decided to keep it around in memory for now, just in case some program wanted it again in the near future. Now, if you instead fetched just those bytes from the file using seek+read or whatever, the OS would treat that request in the exact same way: it'd still read the entire file, and it would still keep the whole thing around in memory. On Linux, you could test this by dropping caches (echo 1 /proc/sys/vm/drop_caches), checking how much memory is listed as free in top, and then using your code to read the same file -- you'll see that the 'free' memory drops by 3 gigabytes, and the 'buffers' or 'cached' numbers will grow by 3 gigabytes. [Note: if you try this experiment, make sure that you don't have the same file opened with np.memmap -- for some reason Linux seems to ignore the request to drop_caches for files that are mmap'ed.] The difference between mmap and reading is that in the former case, then this cache memory will be counted against your process's resident set size. The same memory is used either way -- it's just that it gets reported differently by your tool. And in fact, this memory is not really used at all, in the way we usually mean that term -- it's just a cache that the OS keeps, and it will immediately throw it away if there's a better use for that memory. The only reason it's loading the whole 3 gigabytes into memory in the first place is that you have 3 gigabytes of memory to spare. You might even be able to tell the OS that you *won't* be reading that file again, so there's no point in keeping it all cached -- on Unix this is done via the madavise() or posix_fadvise() syscalls. (No guarantee the OS will actually listen, though.) This is interesting, and on my machine I think I've verified that what you say is actually true. This all makes theoretical sense, but goes against some experiments I and my colleagues have done. For example, a colleague of mine was able to read a couple of large files in using my code but not using memmap. The combined files were greater than memory size. With memmap the code started swapping. This was on 32-bit OSX. But as I said, I just tested this on my linux box and it works fine with numpy.memmap. I don't have an OSX box to test this. I've seen this on OS X too. Here's another example on Linux: http://thread.gmane.org/gmane.comp.python.numeric.general/43965. Using tcmalloc was reported by a couple of people to solve that particular issue. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
On Wed, Feb 29, 2012 at 10:22 AM, Paweł Biernat pw...@wp.pl wrote: I am completely new to Numpy and I know only the basics of Python, to this point I was using Fortran 03/08 to write numerical code. However, I am starting a new large project of mine and I am looking forward to using Python to call some low level Fortran code responsible for most of the intensive number crunching. In this context I stumbled into f2py and it looks just like what I need, but before I start writing an app in mixture of Python and Fortran I have a question about numerical precision of variables used in numpy and f2py. Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I have investigated the float128 data type, but it seems to work as binary64 or binary80 depending on the architecture. If there is currently no way to interact with binary128, how hard would it be to patch the sources of numpy to add such data type? I am interested only in basic stuff, comparable in functionality to libmath. As said before, I have little knowledge of Python, Numpy and f2py, I am however, interested in investing some time in learing it and implementing the mentioned features, but only if there is any hope of succeeding. Numpy does not have proper support for the quadruple precision float numbers, because very few implementation do (no common CPU handle it in hw, for example). The dtype128 is a bit confusingly named: the 128 refers to the padding in memory, but not its real precision. It often (but not always) refer to the long double in the underlying C implementation. The latter depends on the OS, CPU and compilers. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
Hi, Le 29/02/2012 16:22, Paweł Biernat a écrit : Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I googled a bit this __float128. It seems a fairly new addition (GCC 4.6, released March 2011). The related point in the changelog [1] is : GCC now ships with the LGPL-licensed libquadmath library, which provides quad-precision mathematical functions for targets with a __float128 datatype. __float128 is available for targets on 32-bit x86, x86-64 and Itanium architectures. The libquadmath library is automatically built on such targets when building the Fortran compiler. It seems this __float128 is newcomer in the picture of data types that Matthew just mentioned. As David says, arithmetic with such a 128 bits data type is probably not hardwired in most processors (I mean Intel friends) which are limited to 80 bits (long doubles) so it may be a bit slow. However, this GCC implementation with libquadmath seems to create some level of abstraction. Maybe this is one acceptably good way for a real IEEE float 128 dtype in numpy ? Best, Pierre [1] http://gcc.gnu.org/gcc-4.6/changes.html signature.asc Description: OpenPGP digital signature ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
On Feb 29, 2012, at 11:52 AM, Pierre Haessig wrote: Hi, Le 29/02/2012 16:22, Paweł Biernat a écrit : Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I googled a bit this __float128. It seems a fairly new addition (GCC 4.6, released March 2011). The related point in the changelog [1] is : GCC now ships with the LGPL-licensed libquadmath library, which provides quad-precision mathematical functions for targets with a __float128 datatype. __float128 is available for targets on 32-bit x86, x86-64 and Itanium architectures. The libquadmath library is automatically built on such targets when building the Fortran compiler. Great find! It seems this __float128 is newcomer in the picture of data types that Matthew just mentioned. As David says, arithmetic with such a 128 bits data type is probably not hardwired in most processors (I mean Intel friends) which are limited to 80 bits (long doubles) so it may be a bit slow. However, this GCC implementation with libquadmath seems to create some level of abstraction. Maybe this is one acceptably good way for a real IEEE float 128 dtype in numpy ? That would be really nice. The problem here is two-folded: * Backwards-compatibility. float128 should represent a different data-type than before, so we probably should find a new name (and charcode!) for quad-precision. Maybe quad128? * Compiler-dependency. The new type will be only available on platforms that has GCC 4.6 or above. Again, using the new name for this should be fine. On platforms/compilers not supporting the quad128 thing, it should not be defined. Uh, I foresee many portability problems for people using this, but perhaps it is worth the mess. -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Cygwin compile: fatal error: fenv/fenv.c:
Hi all, I am getting the following error when running `python setup.py install` for Numpy in Cygwin. This error happens on the latest as well as the maintenance branched for 1.5 and 1.6. ... creating build/temp.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath creating build/temp.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c' gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/npy_math.c gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. error: Command gcc -fno-strict-aliasing -g -O2 -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c -o build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.o failed with exit status 1 Thanks ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
Pierre Haessig pierre.haessig at crans.org writes: Hi, Le 29/02/2012 16:22, Paweł Biernat a écrit : Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I googled a bit this __float128. It seems a fairly new addition (GCC 4.6, released March 2011). The related point in the changelog [1] is : GCC now ships with the LGPL-licensed libquadmath library, which provides quad-precision mathematical functions for targets with a __float128 datatype. __float128 is available for targets on 32-bit x86, x86-64 and Itanium architectures. The libquadmath library is automatically built on such targets when building the Fortran compiler. It seems this __float128 is newcomer in the picture of data types that Matthew just mentioned. As David says, arithmetic with such a 128 bits data type is probably not hardwired in most processors (I mean Intel friends) which are limited to 80 bits (long doubles) so it may be a bit slow. However, this GCC implementation with libquadmath seems to create some level of abstraction. Maybe this is one acceptably good way for a real IEEE float 128 dtype in numpy ? Best, Pierre [1] http://gcc.gnu.org/gcc-4.6/changes.html ___ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Intel also has its own implementation of binary128, although not well documented (as you said, it's software emulated, but still quite fast): http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/fortran/lin/compiler_f/fpops/fortran/fpops_flur_f.htm The documentation is for Fortran's real(16), but I belive the same holds for _Quad type in C. My naive question is, is there a way to recompile numpy with long double (or just float128) replaced with _Quad or __float128? There are at least two compilers that support the respective data types, so this should be doable. I tested interoperability of binary128 with Fortran and C (using gcc and Intel's compilers) and it works like a charm. The only problem that comes to my mind is i/o, because there is no printf format for _Quad or __float128 and fortran routines have to be used to do all i/o. Paweł ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [Numpy] quadruple precision
On Wed, Feb 29, 2012 at 1:09 PM, Francesc Alted franc...@continuum.iowrote: On Feb 29, 2012, at 11:52 AM, Pierre Haessig wrote: Hi, Le 29/02/2012 16:22, Paweł Biernat a écrit : Is there any way to interact with Fortran's real(16) (supported by gcc and Intel's ifort) data type from numpy? By real(16) I mean the binary128 type as in IEEE 754. (In C this data type is experimentally supported as __float128 (gcc) and _Quad (Intel's icc).) I googled a bit this __float128. It seems a fairly new addition (GCC 4.6, released March 2011). The related point in the changelog [1] is : GCC now ships with the LGPL-licensed libquadmath library, which provides quad-precision mathematical functions for targets with a __float128 datatype. __float128 is available for targets on 32-bit x86, x86-64 and Itanium architectures. The libquadmath library is automatically built on such targets when building the Fortran compiler. Great find! It seems this __float128 is newcomer in the picture of data types that Matthew just mentioned. As David says, arithmetic with such a 128 bits data type is probably not hardwired in most processors (I mean Intel friends) which are limited to 80 bits (long doubles) so it may be a bit slow. However, this GCC implementation with libquadmath seems to create some level of abstraction. Maybe this is one acceptably good way for a real IEEE float 128 dtype in numpy ? That would be really nice. The problem here is two-folded: * Backwards-compatibility. float128 should represent a different data-type than before, so we probably should find a new name (and charcode!) for quad-precision. Maybe quad128? * Compiler-dependency. The new type will be only available on platforms that has GCC 4.6 or above. Again, using the new name for this should be fine. On platforms/compilers not supporting the quad128 thing, it should not be defined. Uh, I foresee many portability problems for people using this, but perhaps it is worth the mess. The quad precision library has been there for a while, and quad precision is also supported by the Intel compiler. I don't know about MSVC. Intel has been working on adding quad precision to their hardware for several years and there is an IEEE spec for it, so some day it will be here, but it isn't here yet. It's a bit sad, I could use quad precision in FORTRAN on a VAX 25 years ago. Mind, I only needed it once ;) I suppose lack of pressing need accounts for the delay. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Cygwin compile: fatal error: fenv/fenv.c:
On Wed, Feb 29, 2012 at 9:10 PM, Matt Miller mattm...@gmail.com wrote: Hi all, I am getting the following error when running `python setup.py install` for Numpy in Cygwin. This error happens on the latest as well as the maintenance branched for 1.5 and 1.6. This should fix it: http://projects.scipy.org/numpy/ticket/1944. Can you confirm that that works? Then I'll make the change in master. Ralf ... creating build/temp.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath creating build/temp.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c' gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/npy_math.c gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. error: Command gcc -fno-strict-aliasing -g -O2 -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c -o build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.o failed with exit status 1 Thanks ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Cygwin compile: fatal error: fenv/fenv.c
That fixed changed my error message to this: numpy/core/src/private/lowlevel_strided_loops.h:404:1: warning: ‘PyArray_PrepareThreeRawArrayIter’ declared ‘static’ but never defined numpy/core/src/private/lowlevel_strided_loops.h:430:1: warning: ‘PyArray_PrepareFourRawArrayIter’ declared ‘static’ but never defined gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o -L/usr/lib/python2.6/config -Lbuild/temp.cygwin-1.7.11-i686-2.6 -lnpymath -lpython2.6 -o build/lib.cygwin-1.7.11-i686-2.6/numpy/core/umath.dll build/temp.cygwin-1.7.11-i686-2.6/libnpymath.a(ieee754.o):ieee754.c:(.rdata+0x0): multiple definition of `_npy__fe_dfl_env' build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o:umathmodule_onefile.c:(.rdata+0x13158): first defined here collect2: ld returned 1 exit status build/temp.cygwin-1.7.11-i686-2.6/libnpymath.a(ieee754.o):ieee754.c:(.rdata+0x0): multiple definition of `_npy__fe_dfl_env' build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o:umathmodule_onefile.c:(.rdata+0x13158): first defined here collect2: ld returned 1 exit status error: Command gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o -L/usr/lib/python2.6/config -Lbuild/temp.cygwin-1.7.11-i686-2.6 -lnpymath -lpython2.6 -o build/lib.cygwin-1.7.11-i686-2.6/numpy/core/umath.dll failed with exit status 1 Thanks for the quick reply! Hi all, I am getting the following error when running `python setup.py install` for Numpy in Cygwin. This error happens on the latest as well as the maintenance branched for 1.5 and 1.6. This should fix it: http://projects.scipy.org/numpy/ticket/1944. Can you confirm that that works? Then I'll make the change in master. Ralf ... creating build/temp.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath creating build/temp.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c' gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/npy_math.c gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. error: Command gcc -fno-strict-aliasing -g -O2 -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c -o build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.o failed with exit status 1 Thanks ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Cygwin compile: fatal error: fenv/fenv.c
More reading of the thread linked solved the issue. To reiterate, add numpy/ and change .c to .h in line 590 of ieee754.c.src. Ex: elif defined(__CYGWIN__) include numpy/fenv/fenv.h endif Thanks, On Wed, Feb 29, 2012 at 1:41 PM, Matt Miller mattm...@gmail.com wrote: That fixed changed my error message to this: numpy/core/src/private/lowlevel_strided_loops.h:404:1: warning: ‘PyArray_PrepareThreeRawArrayIter’ declared ‘static’ but never defined numpy/core/src/private/lowlevel_strided_loops.h:430:1: warning: ‘PyArray_PrepareFourRawArrayIter’ declared ‘static’ but never defined gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o -L/usr/lib/python2.6/config -Lbuild/temp.cygwin-1.7.11-i686-2.6 -lnpymath -lpython2.6 -o build/lib.cygwin-1.7.11-i686-2.6/numpy/core/umath.dll build/temp.cygwin-1.7.11-i686-2.6/libnpymath.a(ieee754.o):ieee754.c:(.rdata+0x0): multiple definition of `_npy__fe_dfl_env' build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o:umathmodule_onefile.c:(.rdata+0x13158): first defined here collect2: ld returned 1 exit status build/temp.cygwin-1.7.11-i686-2.6/libnpymath.a(ieee754.o):ieee754.c:(.rdata+0x0): multiple definition of `_npy__fe_dfl_env' build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o:umathmodule_onefile.c:(.rdata+0x13158): first defined here collect2: ld returned 1 exit status error: Command gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/umath/umathmodule_onefile.o -L/usr/lib/python2.6/config -Lbuild/temp.cygwin-1.7.11-i686-2.6 -lnpymath -lpython2.6 -o build/lib.cygwin-1.7.11-i686-2.6/numpy/core/umath.dll failed with exit status 1 Thanks for the quick reply! Hi all, I am getting the following error when running `python setup.py install` for Numpy in Cygwin. This error happens on the latest as well as the maintenance branched for 1.5 and 1.6. This should fix it: http://projects.scipy.org/numpy/ticket/1944. Can you confirm that that works? Then I'll make the change in master. Ralf ... creating build/temp.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6 creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath creating build/temp.cygwin-1.7.11-i686-2.6/numpy creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src creating build/temp.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath compile options: '-Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c' gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/npy_math.c gcc: build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. numpy/core/src/npymath/ieee754.c.src:590:25: fatal error: fenv/fenv.c: No such file or directory compilation terminated. error: Command gcc -fno-strict-aliasing -g -O2 -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Inumpy/core/include -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/multiarray -Ibuild/src.cygwin-1.7.11-i686-2.6/numpy/core/src/umath -c build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.c -o build/temp.cygwin-1.7.11-i686-2.6/build/src.cygwin-1.7.11-i686-2.6/numpy/core/src/npymath/ieee754.o failed with exit status 1 Thanks ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] YouTrack License
28.02.2012 22:11, Ralf Gommers kirjoitti: [clip] How about just putting it in a new github repo so everyone can see/review the mapping between Trac and YouTrack fields? We should probably create a basic export from YouTrack script at the same time, to make sure there's no lock-in. Keeping the conversion script in the public sounds good. We could perhaps also consider applying hosting for the Scipy tracker, too. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Cygwin compile: fatal error: fenv/fenv.c
On Wed, Feb 29, 2012 at 11:07 PM, Matt Miller mattm...@gmail.com wrote: More reading of the thread linked solved the issue. To reiterate, add numpy/ and change .c to .h in line 590 of ieee754.c.src. Ex: elif defined(__CYGWIN__) include numpy/fenv/fenv.h endif Thanks for confirming. Fixed in master and 1.6.x now. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposed Roadmap Overview
I Would like to hear the opinions of others on that point, but yes, I think that is an appropriate procedure. Travis -- Travis Oliphant (on a mobile) 512-826-7480 On Feb 29, 2012, at 10:54 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Wed, Feb 29, 2012 at 1:46 AM, Travis Oliphant tra...@continuum.io wrote: We already use the NEP process for such decisions. This discussion came from simply from the *idea* of writing such a NEP. Nothing has been decided. Only opinions have been shared that might influence the NEP. This is all pretty premature, though --- migration to C++ features on a trial branch is some months away were it to happen. Fernando can correct me if I'm wrong, but I think he was asking a governance question. That is: would you (as BDF$N) consider the following guideline: As a condition for accepting significant changes to Numpy, for each significant change, there will be a NEP. The NEP shall follow the same model as the Python PEPs - that is - there will be a summary of the changes, the issues arising, the for / against opinions and alternatives offered. There will usually be a draft implementation. The NEP will contain the resolution of the discussion as it relates to the code For example, the masked array NEP, although very substantial, contains little discussion of the controversy arising, or the intended resolution of the controversy: https://github.com/numpy/numpy/blob/3f685a1a990f7b6e5149c80b52436fb4207e49f5/doc/neps/missing-data.rst I mean, although it is useful, it is not in the form of a PEP, as Fernando has described it. Would you accept extending the guidelines to the NEP format? Best, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
