Re: [Numpy-discussion] question about future support for python-3
Darren Dale wrote: I'm not a core numpy developer and don't want to step on anybody's toes here. But I was wondering if anyone had considered approaching the Python Software Foundation about support to help get numpy working with python-3? What kind of support are you talking about? Developers, money, software, PR, test platforms ...? For quite some time we are talking about ways on the PSF list to aid projects. We are trying to figure out what projects need, especially high profile projects and important infrastructure projects. I myself consider NumPy as a great asset for both the scientific community and Python. It's true that Pycon '09 was a major drawback on our financials. But there are other ways beside money to assist projects. For example the snakebite network (http://snakebite.org/) could be very useful for you once it's open. Please don't ask me about details on the status, I don't have an account yet. About a month ago we got 14 MSDN premium subscriptions with full access to MS development tools and all Windows platforms, which is very useful for porting and testing application on Windows. Some core developers may also be interested to assist you directly. The PSF might (!) even donate some money but I'm not in the position to discuss it. I can get you in touch with the PSF if you like. I'm a PSF member and a core developer. Christian ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] parallel compilation of numpy
David Cournapeau wrote: No, and it never will. Parallel builds requires to build with dependency handling. Even make does not handle it well: it works most of the time by accident, but there are numerous problems (try for example building lapack with make -j8 on your 8 cores machine - it will give a bogus library 90 % of the time, because it starts building static library with ar while some object files are still built). You may call me naive and ignorant. Is it really that hard to archive some kind of poor man's concurrency? You don't have to parallelize everything to get a speed up on multi core machines. Usually the compile process from C/C++ file to an object files takes up most of the time. How about * assemble a list of all C/C++ source files of all extensions. * compile all source files in parallel * do the rest (linking etc.) in serial This should give a nice speed up without much work and without complex dependency analysis. Do you see a possible pit fall? I don't. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A buildbot farm with shell access - for free ?
David Cournapeau wrote: It is said in the email that this is reserved to the python project, and prominent python projects like Twisted and Django. Would it be ok to try to be qualified as a prominent python project as well ? Give it some time. Nobody - not even the Python core devs - have access to the machines. It's going to take at least several weeks to get the infrastructure running and to establish a policy. From my perspective NumPy and Sage both count as prominent Python projects. Heck, you are in competition with tools like Matlab and you ain't looking bad! Furthermore you could make better use of the machines than Django because you are using way more C extensions and esoteric libraries. I recommend you subscribe to the snakebite list and bring up your interest. You got my +1 already. For now the list is snakebite-l...@googlegroups.com but it will move to another server (probably Python.org) eventually. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Win64 build?
David Cournapeau schrieb: Do you only need numpy or also scipy ? If you only need numpy, it is relatively straightforward because you don't need BLAS/LAPACK nor any fortran compiler. You should use the Visual Studio compiler, though: VS 2005 for python 2.5 or VS 2008 for python 2.6 - mingw does not work well yet for 64 bits. The offical Windows builds of Python 2.5 are created with Visual C 7.1 (also known as VS2003). You can compile an extension with VS 2005 but that will cause trouble. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] np.nan and ``is``
Andrew Dalke wrote: There are a few things that Python-the-language guarantees are singleton objects which can be compared correctly with is. Those are: True, False, None The empty tuple () and all interned strings are also guaranteed to be singletons. String interning is used to optimize code on C level. It's much faster to compare memory addresses than objects. All strings can be interned through the builtin function intern like s = intern(s). For Python 3.x the function was moved in the the sys module and changed to support str which are PyUnicode objects. So, back to NaN. There's no guarantee NaN is a singleton object, so testing with is almost certainly is wrong. In fact, at the bit-level there are multiple NaNs. A NaN (according to Wikipedia) fits the following bit pattern. NaN: xaxx. x = undefined. If a = 1, it is a quiet NaN, otherwise it is a signalling NaN. The definition is correct for all doubles on IEEE 754 aware platforms. Python's float type uses the double C type. Almost all modern computers have either hardware IEEE 754 support or software support for embedded devices (some mobile phones and PDAs). http://en.wikipedia.org/wiki/IEEE_754-1985 The Python core makes no difference between quiet NaNs and signaling NaNs. Only errno, input and output values are checked to raise an exception. We were discussion the possibility of a NaN singleton during our revamp of Python's IEEE 754 and math support for Python 2.6 and 3.0. But we decided against it because the extra code and cost wasn't worth the risks. Instead I added isnan() and isinf() to the math module. All checks for NaN, inf and the sign bit of a float must be made through the appropriate APIs - either the NumPy API or the new APIs for floats. Hope to shed some light on things Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Ondrej Certik wrote: Are we able to provide an actual patch to Python that implements this? If so, then I am. Imho the proposal should come with an actual patch, otherwise it's difficult to judge it. Your better off with writing a PEP first. In order to implement the proposal you've to make changes to the parser and tokenizer - perhaps to the grammar, too. The code is rather complex and tricky - and not very well documented. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Ondrej Certik wrote: Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Andrew has already pointed it out very well. I like to comment on your proposal from the perspective of a Python core developer as well as the perspective of somebody who has worked with Guido for more than a year. I'd bet my life that Guido is never every going to allow it. The core types are fundemental to the Python interpreter. Even the possibility of pluggable type methods would make the implementation slower, more fragile and much more complicated. We'd have to remove several speed tricks and special cases for e.g. ints and replace them with slower implementations. But don't give up hope yet! During the alpha phase of Python 3.0 and the revamping of the decimal module, some core developers had an even better idea. We were discussing the possibility of making decimals the default for float literals. The idea was rejected eventually but it gave birth to yet another idea. What about making the *result* of a literal pluggable? The Python creates a float for the literal 1.0. Some header in a module could replace the default target 'float' with e.g. decimal.Decimal. Example syntax (rough idea): type(1.0) type 'float' with float as from decimal import Decimal type(1.0) class 'decimal.Decimal' Wouldn't that solve your general problem more elegant without breaking other modules? Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Andrew Dalke wrote: When would this with float ... considered valid? [long posting] Oh h... what have I done ... *g* Slow down, please. For now there are no concrete plans what-so-ever to implement the feature in the near future. Some developers have expressed their interest in a way to alter the resulting type of a literal. It was my attention to show you, that we have discussed the idea, too. Now for the with type as from import syntax. I came up with the syntax idea about an hour ago. I tried to come up with some nice syntax that reuses existing keywords. IMHO it has a nice ring. Other possibilities I came up with: def float as factory def float as from module import factory with float yield factory with float yield from module import factory After some careful thinking I'm in favor of with ... yield It's less ambiguous and can't be mistaken for with open(filename) as fh. The ideas needs a good PEP. You are definitely up to something. You also came up with a list of possible issues and corner cases. Are you interested in pursuing the proposal? *wink* Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Possible new multiplication operators for Python
Andrew Dalke wrote:
Or write B \circledast C ? (Or \oast?) Try using Google to search
for that character.
unicodedata.lookup('CIRCLED ASTERISK OPERATOR')
'⊛'
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C99 on windows
Pauli Virtanen wrote: To clarify this again: *no* features of C99 were used. The C99 specs were only used as a guideline to what behavior we want of complex math functions, and I wrote tests for this, and marked failing ones as skipped. Got it. However, it turned out that different tests fail on different platforms, which means that the inf/nan handling of our complex-valued functions is effectively undefined. Eventually, most of the tests had to be marked as skipped, and so it made more sense to remove them altogether. We hit the same issue during our work for Python 2.6 and 3.0. We came to the conclusion that we can't rely on the platform's math functions (especially trigonometric and hyperbolic functions) for special values. So Mark came up with the idea of lookup tables for special values. Read my other mail for more informations. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Addition of a dict object to all NumPy objects
Robert Kern wrote: I think you could make the dictionary created lazily on the first getattr(). In order to make it work you have to reserve space for a PyObject* pointer for the instance dict somewhere in your type definition. It's going to increase the size of every object by 4 bytes on a 32bit OS or 8 bytes on a 64bit OS, aka sizeof(uintptr_t). An empty dict increases the size of every object by ~30 byte. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Addition of a dict object to all NumPy objects
Robert Kern wrote: Yes, we know that. The concern I was addressing was the time overhead for creating the new dict object every time an ndarray gets instantiated. Most of these dict objects would be unused, so we would be wasting a substantial amount of time. If you push off the creation of the dict to the first time the user accesses it, then we're not wasting any time. We do realize that space for the pointer must still be reserved. I'm sorry for pointing to the obvious. I *guess* it's possible to delay the creation even further. You don't have to create a dict until somebody assigns a new attribute to an instance. It'd require some more code and you'd have to trade memory efficiency for slightly slower access to the additional attributes. Please also note that CPython uses a freelist of unused dict instances. The default size of the dict free list is 80 elements. The allocation and deallocation of dicts is cheap if you can stay below the threshold. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C99 on windows
David Cournapeau wrote: The current trunk has 14 failures on windows (with mingw). 12 of them are related to C99 (see ticket 869). Can the people involved in recent changes to complex functions take a look at it ? I think this is high priority for 1.2.0 I'm asking just out of curiosity. Why is NumPy using C99 and what features of C99 are used? The Microsoft compilers aren't supporting C99 and they'll probably never will. I don't know if the Intel CC supports C99. Even GCC doesn't implement C99 to its full extend. Are you planing to restrict yourself to MinGW32? I'm not a NumPy developer but I'm a Python core developer. I've laid the foundation for the VS 2008 build system for 2.6 / 3.0. Marc Dickinson and I have put lots of work into mathematical, numerical and IEEE 754 fixes. The work was mostly based on the C99 specs but we used C89 code. That should explain my interest in the matter. :] Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] C99 on windows
Charles R Harris wrote: I believe C99 was used as a guide to how complex corner cases involving +/-0, +/-inf, etc. should behave. However, it doesn't look possible to make that behaviour portable without a lot of work and it probably isn't worth the trouble. At the moment the failing tests have been removed. We used the C99 specs as guideline, too. If I recall correctly mostly Annex F and G. We got it all sorted out but it took us a tremendous amount of time and work. We had to reimplement a bunch of math functions like log1p and the reversed hyperbolic functions. Mark introduced a system of lookup tables for complex corner cases. You can find them at the end of the cmath module: http://svn.python.org/projects/python/trunk/Modules/cmathmodule.c The new pymath files contain a series of macros and re-implemenation of C99 features and cross platform workarounds: http://svn.python.org/projects/python/trunk/Include/pymath.h http://svn.python.org/projects/python/trunk/Python/pymath.c HTH Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.arccos(numpy.inf)????
Stuart Brorson schrieb: Hi -- Sorry to be a pest with corner cases, but I found another one. [...] Mark and I spent a *lot* of time in fixing those edge cases in Python 2.6 and 3.0. We used the C99 standard as template. I recommend that you look at our code. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] python memory use
Robin schrieb: If I try to allocate something too big for the available memory I often get a MemoryError exception. However, in other situations, Python memory use continues to grow until the machine falls over. I was hoping to understand the difference between those cases. From what I've read Python never returns memory to the OS (is this right?) so the second case, python is holding on to memory that it isn't really using (for objects that have been destroyed). I guess my question is why doesn't it reuse the memory freed from object deletions instead of requesting more - and even then when requesting more, why does it continue until the machine falls over and not cause a MemoryError? Your assumption isn't correct. Python releases memory. For small objects Python uses its own memory allocation system as explained in http://svn.python.org/projects/python/trunk/Objects/obmalloc.c . For integer and floats uses a separate block allocation schema. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Compile Numpy in VC++8
Matthieu Brucher schrieb: Hi, As I've said, you must start by compiling Python with VC++ 8, that means using the 2.6 alpha. Negative Houston Python 2.6 and 3.0 are using VS 2008 aka VC 9.0 Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Handling of numpy.power(0, something)
Stuart Brorson wrote: I have been poking at the limits of NumPy's handling of powers of zero. I find some results which are disturbing, at least to me. Here they are: [SNIPP] Please checkout Mark Dickinson's and my trunk-math branch of Python 2.6. We have put lots of effort into fixing edge cases of floats, math and cmath functions. The return values are either based on the latest revision of IEEE 754 or the last public draft of the C99 standard (1124, Annex F and G). For pow the C99 says: math.pow(0, 0) 1.0 math.pow(0, 1) 0.0 [30859 refs] math.pow(0, float(inf)) 0.0 math.pow(0, float(nan)) nan math.pow(0, -1) Traceback (most recent call last): File stdin, line 1, in module ValueError: math domain error Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Handling of numpy.power(0, something)
Stuart Brorson wrote: math.pow(0, -1) Traceback (most recent call last): File stdin, line 1, in module ValueError: math domain error Why isn't this one inf? The standard says return inf and raise a divide-by-zero floating point exception. Since we can't do both in Python we sticked to the exception part. Also, what do these specs say about 0^complex? See for yourself http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf The interesting information are in Annex F.9 and Annex G.6. So far we haven't dealt with complex powers and Python doesn't support 0.**1j yet. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?
Matthieu Brucher wrote: When Visual Studio 2008 will be used, there might be a way of using the manifest files (that were created for a similar purpose). For the moment, All I know is that you must put the dll in the Windows/system32 folder or somewhere in the PATH. That's not enough for some DLLs. .NET assemblies as well as Side-by-Side dlls (SxS) must be registered properly. You can install a SxS dll in PATH but simple copying the DLL isn't enough. It also depends on the OS. Have fun! The SxS issue has bitten us in the a.. when we delivered the second beta of Python 3.0. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?
David Cournapeau wrote: Do you have a link to the related python ML discussion by any chance ? No, I'm sorry. It was a private chat between between Guido, Martin and me during the release phase of Python 3.0a2. The MSDN website has some articles about SxS DLLs though. I had to read about ten articles to get the big picture. The information is scattered all over the place. :/ Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Float and math module enhancements for Python 2.6 and 3.0
Hello! The Python core developers are currently working on several improvements for floats and the math module. Since you are power users I like to get your opinion and suggestions on several patches: http://bugs.python.org/issue1534 -- sys.float_info [done] http://bugs.python.org/issue1580 -- shorter repr of floats http://bugs.python.org/issue1635 -- platform independent representation and creation of +/-inf and nan http://bugs.python.org/issue1640 -- additional functions for the math module new: sys.maxsize, gone in 3.0: sys.maxint Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
