Re: [Cython] Can't push to cython-devel
I'm a little late to this party, it seems, but I wanted to throw in my vote. 1. The web site 2. Trac 3. Wiki 4. Repositories 5. Buildbot 6. Mailing lists I think these + code review make for a pretty good list, and I'd fight hard to claim code review integrated with a bug tracker should be a requirement. Personally, I'm a big +1 on moving everything to either github or code.google.com, which seems like the best of the choices. If people really think the geographic restrictions are a problem, I guess I'm fine with dropping code.google.com off the list -- but just for the sake of completeness, are there any features code.google.com has that github doesn't? (Or vice-versa?) Are they things we'd ever be likely to notice? I'm also more than happy to switch to git for Cython, especially in light of the hg-on-git stuff Dag mentioned. My sense is that github definitely seems to be the new hotness -- I'll admit that there's something about this I don't get, but if that just means more people might notice Cython, I'd say that's a win. I have one or two questions to put out there: * Does github have any sort of mailing list support? I'd love to see cython-dev migrated to something with better searchability, linking, archives, etc -- I don't know that there's any special link between google groups and code.google.com, so this is really orthogonal to the project hosting question. I know the decision to use codespeak was made as an explicit decision against google groups, but my understanding is that whatever objections there were are now gone. Is that still the case? * Has anyone actually been an admin on a github project? The tools sure sound good from the outside, but are they fairly pleasant to use? Trading writing our own apache configs for regularly fighting with, say, flaky web interfaces is decidedly *not* a win. (I've done some minor admin on a code.google.com project, and generally had good experiences.) * Are there a bunch of examples of projects that host basically everything out of github? I'd be curious to check one out, just for the sake of it. * I do have one minor worry about github, which may have something to do with me not understanding why it's such a phenomenon. The whole model seems to really take the D in DVCS to the extreme -- so much so that I often have a hard time deciding what the central repo for a project is (in cases where there is such a thing) just based on what I see on github, or how to get there when I happen to end up in a random person's branch (say from a google search). My vague impression is that the model for code.google.com is project-centric, whereas the github model is much more developer-centric. Is there something I'm missing? -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Can't push to cython-devel
argument, which is what I think I said above. I generally loathe political debate, and I think cython-dev is a wildly inappropriate place for it. I'm really confused here: I've agreed from the beginning to eliminate what I thought was a reasonable candidate from the running because you had an issue with it, with *no argument whatsoever*. I explained clearly that I don't want to get into a political debate, because I don't think our personal politics have anything to do with what features github has that code.google.com doesn't. Last I checked, I was entitled to my opinions just as much as you are to yours. I really don't get how spouting political rhetoric seemed like it was really going to add to this conversation in any way. This has been an excellent reminder of why I try to avoid getting involved in mailing list threads in the first place -- as always, I regret sending the first email at all. Thanks for helping to keep my experiences consistent. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] [cython-users] String comparison
I think it's ok to consider this specific case a bug. If a user wants pointer comparison, is is the most explicit operator w.r.t. Python semantics. Using strcmp() for the == operator on char* values makes sense to me. We special case char* all over the place, so this is just another exception. Personally, I'm +1 on considering this a bug, but I'm a big -1 (basically -as_many_as_my_vote_is_worth) on changing the behavior of comparison operators on some/all pointer types to fix it. As I understand, the chain of events causing the problem is this: - user writes code with == on Python (bytes) objects, expecting Python semantics - we do type inference and replace bytes with char * - we run into trouble because == already has a different semantics for char * objects. I think the change that we should make is in the second step there, not the third -- I think we're throwing away valuable type information by replacing Python bytes object with char *, so that's the place to fix it. I think we should just introduce a new type which represents a bytes object that we're modeling with a char *, which we can call whatever -- maybe InferredCharStar. It can simply be a wrapper around a char * -- in fact, it can just *be* char * in the generated C/C++ code -- but it maintains Python semantics in any Python statement. We don't even need to expose it to the programmer if we don't want, since it only really exists at compile-time, though I'd probably be in favor of exposing it. (I'm a long-time Scheme and Python programmer that loves types ... I wonder if I should seek help for that. :P) Then there's also no need to introduce any more special cases on char * behavior, or by extension any other C types. This closely mirrors the situation with trying to use a C int for a Python int without changing the semantics, and probably runs into some of the other weird and thorny issues. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Changes from 0.12 to 0.13
Hi Stefan, AttributeError: 'UnspecifiedType' object has no attribute 'create_from_py_utility_code' (full traceback below) Have you seen this before? Unfortunately, yes. It means that the type inferencer left something in a halfway state (i.e. it didn't finish inferring types everywhere). I'd love to see the code (I dealt with dozens of these during the 0.13 release cycle), but as a temporary workaround you can either (1) add explicit type info for the variable it's complaining about, which may involve a little digging, or (2) turn off type inference completely. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Changes from 0.12 to 0.13
Hi Stefan, So here's an ugly answer: put the `cimport numpy as np` before the `import numpy as np`. I've got clean up the semantics of import vs. cimport on my todo list, but it's a big ugly ball of sadness, so it's still sitting on the todo list. ;) Probably the first fix is to write a simple pass that moves the cimports up ... I didn't actually compile the resulting C code (I don't have numpy on this machine), but let me know if that just generates bad C code. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] ANN: Cython 0.13 released!
It is with *great* pleasure that I email to announce the release of Cython version 0.13! This release sets another milestone on the path towards Python compatibility and brings major new features and improvements for the usability of the Cython language. Download it here: http://cython.org/release/Cython-0.13.tar.gz == New Features == * Closures are fully supported for Python functions. Cython supports inner functions and lambda expressions. Generators and generator expressions are __not__ supported in this release. * Proper C++ support. Cython knows about C++ classes, templates and overloaded function signatures, so that Cython code can interact with them in a straight forward way. * Type inference is enabled by default for safe C types (e.g. double, bint, C++ classes) and known extension types. This reduces the need for explicit type declarations and can improve the performance of untyped code in some cases. There is also a verbose compile mode for testing the impact on user code. * Cython's for-in-loop can iterate over C arrays and sliced pointers. The type of the loop variable will be inferred automatically in this case. * The Py_UNICODE integer type for Unicode code points is fully supported, including for-loops and 'in' tests on unicode strings. It coerces from and to single character unicode strings. Note that untyped for-loop variables will automatically be inferred as Py_UNICODE when iterating over a unicode string. In most cases, this will be much more efficient than yielding sliced string objects, but can also have a negative performance impact when the variable is used in a Python context multiple times, so that it needs to coerce to a unicode string object more than once. If this happens, typing the loop variable as unicode or object will help. * The built-in functions any(), all(), sum(), list(), set() and dict() are inlined as plain `for` loops when called on generator expressions. Note that generator expressions are not generally supported apart from this feature. Also, tuple(genexpr) is not currently supported - use tuple([listcomp]) instead. * More shipped standard library declarations. The python_* and stdlib/stdio .pxd files have been deprecated in favor of clib.* and cpython[.*] and may get removed in a future release. == Python compatibility == * Pure Python mode no longer disallows non-Python keywords like 'cdef', 'include' or 'cimport'. It also no longer recognises syntax extensions like the for-from loop. * Parsing has improved for Python 3 syntax in Python code, although not all features are correctly supported. The missing Python 3 features are being worked on for the next release. * from __future__ import print_function is supported in Python 2.6 and later. Note that there is currently no emulation for earlier Python versions, so code that uses print() with this future import will require at least Python 2.6. * New compiler directive language_level (valid values: 2 or 3) with corresponding command line options -2 and -3 requests source code compatibility with Python 2.x or Python 3.x respectively. Language level 3 currently enforces unicode literals for unprefixed string literals, enables the print function (requires Python 2.6 or later) and keeps loop variables in list comprehensions from leaking. * Loop variables in set/dict comprehensions no longer leak into the surrounding scope (following Python 2.7). List comprehensions are unchanged in language level 2. == Incompatible changes == * The availability of type inference by default means that Cython will also infer the type of pointers on assignments. Previously, code like this cdef char* s = ... untyped_variable = s would convert the char* to a Python bytes string and assign that. This is no longer the case and no coercion will happen in the example above. The correct way of doing this is through an explicit cast or by typing the target variable, i.e. cdef char* s = ... untyped_variable1 = bytess untyped_variable2 = objects cdef object py_object = s cdef bytes bytes_string = s * bool is no longer a valid type name by default. The problem is that it's not clear whether bool should refer to the Python type or the C++ type, and expecting one and finding the other has already led to several hard-to-find bugs. Both types are available for importing: you can use from cpython cimport bool for the Python bool type, and from libcpp cimport bool for the C++ type. == Contributors == Many people contributed to this release, including: * David Barnett * Stefan Behnel * Chuck Blake * Robert Bradshaw * Craig Citro * Bryan Cole * Lisandro Dalcin * Eric Firing * Danilo Freitas * Christoph Gohlke * Dag Sverre Seljebotn * Kurt Smith * Erik Tollerud * Carl Witty -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] this link appears to be broken
Thanks. We're in the process of doing a release and re-generating the docs, so hopefully that link will be up and working again soon. Yep -- our script doesn't rebuild the latex/pdf files by default, and trying to run that just now broke. I'm going to look into it some time today. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Cython performance bug for special functions
Hi Stefan, Ah, got it. This is just what Carl wrote in the initial bug report. The new setup fails to provide a __getattr__ method and implements a __getattribute__ method instead. But the problem is not only that the introspection says it's not there, the problem is that the method really is not there. So we break user code that tries to use __getattr__ as an unbound method *and* the code generated by Cython for setting and retrieving docstrings of this method, thus breaking the module initialisation. So it's a double regression that is clearly user visible. Sounds like a release blocker to me. Robert and I were chatting about this earlier -- did the changes you pushed completely fix this problem, or just cover it up? We couldn't come up with a case that was still a problem off the cuff, but the commit message said something like for now, which made us think that there might be more to the story. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] cython-release or cython-devel?
Perhaps you should remove cython-release? Actually, the plan is to use it tonight to generate an rc. At that point, we'll actually want the separation -- new stuff should go in devel, and release should stay nice and working. ;) I think it makes sense to have a branch that basically only gets used around release-time ... -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] Cython 0.13.beta1
Hi all, So here's a second go at Cython 0.13 beta. I'm posting it to the usual place, namely: http://www.cython.org/release/Cython-0.13.beta1.tar.gz This definitely still isn't ready to go out the door, hence the beta instead of rc. A few notes: * We've been running into an interesting problem with rare (~0.05% of the time) segfaults on Sage startup. Interestingly, this has apparently been around a while -- Robert and I have run a handful of tests, and this behavior was already there with Cython-0.12.1. (We didn't try to bisect at all, we just wanted to confirm that it wasn't completely new.) Our not-completely-scientific analysis suggests that the rate of segfaults hasn't changed since the last Cython release, so our current plan is to get this release (finally!) out the door, and then start trying to diagnose this further. There are some questions about whether or not this might be hardware-related, but given how rare the failures are, I don't think we've really run the tests anywhere enough to say we're *not* seeing the same problem elsewhere. * Since it seems like every third email to cython-users has confusion about bool being a C++ vs. Python type as its root cause, Robert's gone ahead and made bool an invalid type identifier. (This is already pushed, so this might not be news.) The plan is to simply make the types explicit for now (which Robert has already done, see cimport cpython.bool and libcpp.bool) and decide later whether or not bool should be a valid identifier by itself. I'm going to add a FAQ entry about this, but I'm going to do it when I'm more coherent (because writing FAQ entries takes more concentration than cutting a beta?). Things we need to fix before release: * There are a handful of unicode-related doctest failures that Lisandro ran into in another thread. I'm guessing Stefan will know what's going on here faster than anyone else? ;) * We need to clean up the last of the fallout from #561 -- if I'm reading the corresponding thread correctly, Carl's got a fix for that we need to push. If so, Carl, feel free to push that. (My machine is currently rebuilding Sage, so I can't test this myself.) Is there anything else that anyone can think of that's blocking us on 0.13? If not, my plan is to cut an rc as soon as we get all the above failures passing, and plan on releasing a few days after that. If anyone has any other thoughts, now's the time. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Cython 0.13.beta1
As far as I know, I don't have write access to the Cython repositories. E-mail me an .htpasswd file and I'll fix that. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] cython-closures tip passing all Sage doctests!
Hi all, I think the subject line pretty much says it all -- I just successfully built sage-4.4.4 against the tip of cython-closures, and it builds, runs, and passes all doctests(*). As far as I'm concerned, that means it's time to roll ourselves an alpha. (Robert, do you want to tell me what steps are involved in that? Is there a list somewhere?) Also, what's the general rule for avoiding someone clobbering the working state with a push? Should I just add a tag for 0.13-alpha0 and we could always recover from there? I just ran the Cython tests in the closures branch itself -- it looks like there are a few doctest failures with the recent code there. I'm happy to start helping with these tomorrow, but can we designate cython-closures as bugfix only until the actual release? For that matter, with closures becoming devel, soon we won't need a closures branch anymore ... finally. ;) (*) Actually, there's one doctest failure going on -- but it's a bug in Sage, which I've reported. (http://trac.sagemath.org/sage_trac/ticket/9524) It could be that there's an associated underlying Cython bug, but I don't think so in this case -- either way, it shouldn't stop us from rolling an alpha, and a release if we still think it's not an underlying Cython bug. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] Type inference and memory management
Hi all, I ran into an interesting question while getting Sage to build and pass tests against the current cython-devel tip (side note: it does on both my laptop and sage.math!), and I thought I'd get some opinions on the right way to fix it. Consider the following bit of code: def __repr__(self): cdef char* ss = fmpz_poly_to_string(self.poly) s = ss free(ss) return s You don't need to know what an fmpz_poly is -- the relevant fact is that it's a call into some C library, and it returns a string representation in the form of a char *. This code then does what you expect: gets the value from the C library and returns it. The key is that it's using the fact that variables implicitly default to Python objects here -- the s = ss line will cause the underlying char * to be copied into a variable whose memory is managed by Python. I think this is considered the Cythonic way of doing things -- at least, there are entries in the FAQ suggesting that this is a good idea. Now enter type inference. It looks at this block and says, hey, s is only ever assigned to a char * -- let's call it a char *, too. Of course, this is a disaster -- it changes the semantics of the all-important s = ss line. As a result, the return value is (a Python copy of) some random junk. This is easy enough to fix -- we can be more explicit about our intentions, and declare s to be an object, which works great. However, this is likely to break at least some user code in the wild -- especially since we've been recommending this as the right way to do things. I can see at least a few options: 1) Break the code above, tell people to explicitly declare things to be objects. 2) Decide that if a variable gets returned by a function which is either a def'd function or returns a Python object, and we don't have an explicit type declaration already, then we only infer something which is a subtype of Python object. (Right now, we almost never infer anything more specific than Python object anyway.) 3) Something else in between, i.e. make some decisions based on the type that gets inferred. Or, better yet, do some control-flow analysis/alias analysis and start shadowing the variable with a Python object only once you need to. (Clearly this is a pipe dream right now.) I'm personally leaning towards (2) with the hopes of one day getting closer to (3). I've got a patch written that does (2), which I'm happy to push if people agree on that one. I think the tradeoff between (1) and (2) is fairly standard -- (1) is going to generate (potentially) faster code, but (2) is going to be much friendlier for someone migrating Python code. Thoughts? -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] Testing import/cimport and tests spanning nested directories
Hi all, So I'm currently trying to write some tests for a subtle issue involving resolving imports and cimports. (I think I emailed about this a while back.) The particular issue isn't so important right this minute -- the only thing that matters is that to test it, I need a test with a few layers of directories, with __init__.py files sprinkled in several places, and potentially *not* at the root. Unfortunately, this doesn't work so well with the current testing setup -- at least, as I understand, there's no way to tell it when you try to run this test, also copy along this particular chunk of the current directory. So there are two obvious choices: 1) Give a test a way of specifying what other files should come along as part of it. 2) Create a directory called nested (or something else) somewhere in the tests/ tree (maybe one for each of the current subdirectories?), and add code to runtests.py to handle that whole tree at once. Each of these has its ups and downs -- but I'd love it if someone told me that the testing code already did what I'm looking for, and I just didn't know it. ;) I think (1) feels like the classier solution, but (2) would make it easy for someone to *add* a nested test without having to create a whole bunch of files/directories themselves. That said, we'd have to be careful that no one accidentally broke a test by modifying it -- tests really should be nearly immutable. Any thoughts? -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Testing import/cimport and tests spanning nested directories
If this ends up being coded from scratch, I think it would make testing easier if one could actually write down all the files in a single testcase bundle file (e.g. with a test suffix). Something like the following, which is simply split up into seperate files by runtests.py: I *really* like this idea -- I've already gotten tired of browsing through the few files in the tests I've got. ;) Then we wouldn't need to worry about having an extra tree floating around, and it's very natural for someone to copy such a test to use as a base for what they want to do ... +1 -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Cython 0.13 status: coming along nicely.
Yep, I'm going to work on getting everything pushed right now, so that I'm not the only one who can look at this stuff. So I didn't get to add tests to everything, but I'm about to head to bed. Instead of just disappearing into the night, I thought I'd put the patches somewhere people can pull and play with them -- I've got a repo up here: http://sage.math.washington.edu/home/craigcitro/cython-0.13-working/ There are also several changes to the Sage library (which Robert asked about above). These apply cleanly in order against sage-4.4.2. You can find them here: http://sage.math.washington.edu/home/craigcitro/cython-0.13-working/sage-patches/ As you'd expect, the patches are subsumed by the .hg bundle. (Robert, I'll do the work of making tickets for these and cleaning up commit messages tomorrow, once other stuff is ready to go.) So as soon as I've got tests written for all of these, I think everything can get pushed -- and cut the cython-0.13 alpha I promised previously. :P After that, I think it's worth trying to make a quick push to sort out the last issues with sage + the closures code -- I don't think it's in bad shape, and with several of us finally having some time to work again, I'm hopeful that we can get everything fixed up fairly quickly ... but we've seen where my optimism has gotten us in the past. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Cython 0.13 status: coming along nicely.
So, what's the status here? I'd love to know too. Yeah, I'm sorry -- I've totally been terrible about actually replying to email, which is bad form. Here's the status: I decided to try seeing what the state of affairs was with just moving my current stack of patches over to cython-closures. It turns out it's not so bad, but there's still some trouble with type analysis/inference, not related to closures, but instead to the other code in the closures branch. I'm going to sit down right now and clean up these patches and the patches against sage and push, so that it's no longer a mystery where we're at. Craig, you have fixes to many of these, right? (It looks like they're mostly inferred str and undeclared type imports.) I don't want to duplicate effort, but would like to dive into this. Yep, I'm going to work on getting everything pushed right now, so that I'm not the only one who can look at this stuff. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Performance difference?
Pyrex will do this if you declare the object as being of type 'dict' (and it will check the type on the boundary between Python and extension code, so it's still type-safe). Similarly for 'list' and a few other built-in types that have dedicated C APIs. Sure, Cython will also (usually) do this if you *explicitly declare* the type -- any places where it doesn't qualify as TODOs in my head, anyway. In fact, now that we have some type inferencing going on, Cython will even do this in some cases where you don't declare the type, but we can deduce it. In the example above, though, notice I said a function that takes an arbitrary Python object -- I'm talking about the case where there *is* no explicit type information, but you still want to avoid typechecks and dispatching into generic code. I'm talking about a snippet like this: cdef void foo(object x): if isinstance(x, dict): x['spam'] = 5 Is Pyrex smart enough to infer that x is in fact a dict, and emit a call to PyDict_GetItemString or the like? I don't think Cython is right now, but I sure would like to make it smart enough to handle that case. Similarly, if someone made a call to a library function that always returned a dict, it'd be nice to tag the resulting object as a dict for the same kind of things. I think there are at least three key ingredients to handle things like the above: (1) type inference, (2) control-flow analysis, and (3) potentially shadowing variables (if x is used for several different Python types in a block). These are all in the vein of we don't do them yet, but we want to. That said, the real black magic I was referring to is beyond that example: I'm talking about code that *doesn't* have explicit type declarations, and *doesn't* do any type testing -- just blindly takes an arbitrary PyObject * and calls off to a PyDict_* method, or manually reaches into the underlying PyDictObject struct and does something crazy. Of course, this isn't something you can do from Python -- but it *is* something you could do from C. These are generally horribly unsafe things to do, which is what I was saying about segfaults above -- however, there are times that speed is more important than any sort of safety, and that's a barrier you can't cross in Pyrex/Cython without directly using the Python/C API yourself, I think. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Performance difference?
Pyrex will do this if you declare the object as being of type 'dict' (and it will check the type on the boundary between Python and extension code, so it's still type-safe). Similarly for 'list' and a few other built-in types that have dedicated C APIs. Right, this is totally not black magic. Of course not -- but as I mentioned above, this isn't what I was referring to. I was referring to cases where either (1) there's no *explicit* type information, which I think we can handle many of as Cython gets smarter, and (2) cases where it's not *possible* to infer the type information, but you have some sort of implicit contract (because, say, you're writing the code that calls this C extension). There are some cases, though, where you can easily write C-API code that outperforms Cython code. A well known example is a call to PyDict_GetItem() in C and a subsequent check for a NULL return value. Cython code needs to read the value into a variable first and then test it, a common idiom being result = d.get(key) if result is not None: ... Maybe I'm being too optimistic, but there are some cases like this that we *could* teach the compiler to figure out, right? It would probably involve *some* amount of Python-specific analysis, and we surely couldn't catch *all* of the cases, but we could probably get the lion's share of the idiomatic ones ... But in general, this is really chasing microseconds in places where it rarely matters. Sure. But that said, if you *really* do care about those last few microseconds -- because maybe you're obsessed with irrelevant microbenchmarks like I've often been? ;) -- there is a little extra you can squeeze with direct use of the Python/C API. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Performance difference?
What's the size of the performance difference between Cython and a C Extension Module? Obviously this depends on how well each is written, and how many concessions to maintainability are made, but just as rough estimates? If it's not a teensy difference, what is the main thing that accounts for that difference? Is it perhaps the implicit memory management? I don't think this is something you could give a clear-cut answer to without knowing what you were doing in your C extension module, but I'd say the answer should be something along the lines of probably negligible, unless you were going to perform crazy black magic in your C extension. And even if you were going to do something crazy -- you could always create a first pass with Cython, and then tweak the parts where the speed was that essential. For instance, if you're planning on interfacing with a C library, and creating/manipulating some C/C++ data structures, then you should get basically the same performance (unless you got tricked into converting back and forth between C and Python, but avoiding that should fall under the how well each is written bit in your question). If you're planning on mostly working with Python objects in a fairly safe way, then the answer is probably there's still a small gap, but we're working on closing it fast. There are definitely situations where we have enough information to do things like avoid typechecks and generate tighter code, but I don't think we always take advantage of everything that's available. We also do a lot of things you probably wouldn't take the time to do yourself -- for instance, we generate custom argument parsing code, which some people have definitely seen some performance improvements from. However, there's one class of things you could do directly in a C extension that we don't, which is cheat: you could write a function that takes an arbitrary Python object, but decide that you know it will only ever get called with a dict, and directly dispatch to various PyDict_* macros, or access parts of the struct directly. Or you could decide some object doesn't get reused, and just side-effect it instead of allocating a new object to store a result, that kind of thing. You could definitely create some faster, more fragile code this way -- you'd start hitting lots of segfaults if your assumptions were off. (This was the black magic I referred to above.) Maybe that's not directed/helpful enough -- if you give us an idea of what kind of C extension you were thinking about writing, we could probably give a much better answer about why Cython is the right tool for the job. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] Cython 0.13 status: coming along nicely.
Hi all, So I hope you're sitting down for this one: I just ran a complete build and test of Sage with a recent cython-devel + some extra patches, and everything builds and passes. Huzzah! I just need to add some tests to my patches tomorrow and I'll push those, along with posting a few patches to the Sage library. (Robert, you're going to have a few easy reviews coming up! :P) Here's my current plan: - Saturday: add tests, push some patches. (I'm flying to CA tomorrow, which is a perfect time for this.) - Sunday: cut cython-0.13.alpha0, *finally*. After that, I'm going to pull the new patches into cython-closures, and test Sage with that. Here are some famous last words: I don't expect to run into too much trouble, because nothing is *using* the new features. If that goes well, I'll cut alpha1 shortly thereafter. (There's some other code here and there, but at least I'm familiar with parts of it already.) If not, we'll see how bad the fallout is and make a decision. My goal is to get alpha1 out and give people, say, a week or so to try it out, see what trouble they run into, and then plan on doing an actual release shortly thereafter. Thoughts? Comments? What's the schedule usually like? For the curious: there was nothing terribly bad about this particular release -- most of the holdup was due to other things (like buying a house, moving, designing a new course this past quarter, baby illnesses, that kind of stuff). There was one significant change from 0.12.1, though -- safe type inference is turned on by default. As you might expect, this causes *lots* of small changes, especially in a codebase like Sage. So there were a bunch of bugs to track down -- most of which weren't good for hunting 30 minutes at a time. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Status of closure
Hi Leon, I have been following the dev mailing list and got the impression that closure will likely to be merged to the next release. Is it still the case? If this is the case, what branch should I pull to test-run the closure code? I have some mount of code that make heavy use of itertools and would like to test the new features against my code. The plan is still to include closures in 0.13 -- it's currently long-delayed, but it will happen. I'm the release manager, but I've been horribly busy with the rest of life. In the interim, you can just clone the closures branch: http://hg.cython.org/cython-closures/ which also has the most recent cython-devel merged, too. Give it a go, and definitely let us know if you run into any troubles. Warning: closures are supported, but generators/yield are still not, unless Stefan's been doing more than I think. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Haoyu Bai's application accepted for GSoC
Just a suggestion, take it or leave it: For my GSoC work I found google's codereview to be a very helpful tool. http://code.google.com/p/rietveld/wiki/CodeReviewHelp It allows you to submit patches that are presented in an app-engine GUI. Reviewers can make comments suggestions that are local to each change in the patch. Perhaps it would be easier to use for interactive patch review than trac. (It works seamlessly with mercurial, too). +10 -- we should *totally* start using rietveld. (Plus, in a month or two, Robert and I will be totally in love with it.) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Haoyu Bai's application accepted for GSoC
The list of accepted applications for Google Summer of Code 2010 were just announced yesterday, and Haoyu Bai's Cython application was officially accepted. Congratulations, and we're looking forward to a lot of great stuff happing this summer. Happy to read this. Congratulations! Ditto -- between this and several of us having other responsibilities off our shoulders, it should be an exciting summer for Cython. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] rss/atom feeds
Hey all, I don't know if anyone but me had ever run into it, but the RSS/Atom feeds on hg.cython.org were slightly broken -- the URLs they provided for changesets were all referring to http://sagemath/; instead of http://hg.cython.org/; (this is an artifact of the way the sites are hosted). Anyway, I tried to fix this via apache, but that was dicey for various reasons. I fixed this another (slightly more hackish) way -- I think it should be fine (my random testing shows no trouble, but that doesn't mean much). If anyone runs into problems with the hg web service, or hg.cython.org at all, give me a shout. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] good news: cython-closures is all green again
Hey Stefan, I managed to track down the ref-counting bug in the cython-closures branch. Function arguments that participated in the closure were not always INCREFed, which induced several problems with the cleanup of the closure class instance as it expected that any non-NULL field could safely be DECREF-ed. Especially errors during argument conversion were pretty deadly. Sweet! That's interesting ... I won't get time to sit down and look at the patch until tonight at the earliest, but I'm looking forward to seeing what the issue was. After this fix, all tests in the cython-closures pass, and even the failures in the Python regression suite start to get interesting. So, unless there are still objections from others, my vote is for getting it merged back into mainline (it's already up-to-date with cython-devel) and pushing out a 0.13 alpha. (Last I heard, Sage was still failing to build, but that doesn't have to keep us from getting an alpha version out, does it?) This sounds reasonable enough, with one caveat: if something in the closures branch causes huge issues with Sage, we'll either need to keep delaying 0.13 or revert. I can't imagine why it would, but those are famous last words. :) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Wiki is down?
Thanks! This has hit us before too. I think surge protection might have been triggered on the IP, and since it's sitting in a VM behind a proxy server, surge protection, if triggered, impacted everyone. Yep -- I think surge protection was turned off on the Sage wiki for the same reason (i.e. more trouble than it's worth with the VM routing). I think they're still getting more traffic than us by a bit, so they can be our canary. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Question on generator support
Any ideas on when generator support is going to be available? On trac, it's classified under the 1.0 milestone release. Is it safe to assume we are not going to have generators in the 0.13 release? I think it's a safe bet that generators won't be in 0.13 -- I'm still hopeful that closures will be, but that's still a toss-up. I think that there's going to be a flurry of activity come late May/June -- Robert will be done with his PhD, and I'll be done teaching/moving, and I think Dag and Stefan will also have more free time coming soon. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Wiki is down?
The wiki is also down for me -- I received the surge protection message on my first visit today and every attempt thereafter. I just fixed this -- in fact, I turned off surge protection for good. If anyone has more trouble, let me know. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Allow analyse_types() to replace the current node?
What do you think? +1. I don't think it really interferes with the idea of transforms; it's more like going only half the way because it is less likely to break the code. I'm also +1 on this -- in general, I'm happier to see the tree get modified when you know it needs to be rather than try to leave a note and do the right thing later. Also, there currently seems to be a mix of behavior for analyse_types (i.e. most seem to just use side-effects to get the job done, but a few return values) -- it would be nice if these ended up being uniform. If somebody is preparing 0.13 for release then that somebody will have to comment on whether it should wait until after the release though. I'm currently actively working on the 0.13 release. Stefan, have you started already writing the code? If not, or if you're just starting, I'd say we should wait until after the release ... -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] GSoC project for improving Python compatibility
Hi Haoyu, Now my proposal is on the Cython wiki for comment: http://wiki.cython.org/HaoyuBai/GSoC2010 Sorry about the delay -- I'm finally getting some cycles to sit down and work on this. First, minor edits -- I went ahead and edited the page on the wiki and made minor grammatical/spelling/formatting changes. I also left a few notes where I thought you could use more explanation and/or a link -- just search for -cc on the page. (We could also renumber the references, but that's not really a big deal.) I had some thoughts along the same lines as Robert and Stefan for the first part of the project -- I think that there will probably be a little extra time needed at the beginning acclimating to the codebase. (Though maybe the fact that you've already got two other patches up and waiting for review suggests otherwise? ...) Also taking into account Robert and Stefan's advice on ordering of events and how much time things will take, I'd recommend slightly restructuring things, as follows: * Weeks 1-3: bug fixes as-is, making sure to get each passing tests and posted on trac before moving on to the next one * Week 4: Work on pure Python mode documentation, start pinging people for code reviews on patches remaining from weeks 1-3 * Week 5: respond to any comments on patches, and make getting as many patches as possible merged the goal for midterm review. * Week 6-7: decorator support (note that some of the documentation work was moved up) * Week 8-9: annotation support (I'm taking into account Stefan's comment that this is not as deep as you might expect at first) * Week 10: cleanup, documentation, tests, etc. Basically, I want to really focus on getting a bunch of patches and documentation knocked out in the first half, for two reasons: first, as your mentor, I can be reviewing your patches and responding as a way of monitoring your progress, and second, it means that you'll get more familiarity with the current code and documentation before jumping into the slightly-larger-scale decorator and annotation projects. It also means that we have a little extra padding at the beginning for things that take extra time (like Stefan predicted for #465). Does that seems sensible? Feel free to tell me if you think I'm crazy, or if you'd rather do it a different way. I'm going to go work on finishing up whatever else I have to do for mentor registration, but I should be able to be fairly responsive by email for the rest of the week ... and I'm getting excited about this project! -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] semantics of import and cimport
I have a question about the semantics of mixing imports and cimports. I started drafting a long message with lots of details and examples, but I realized I should just start with a few simple questions: if you do from Foo import Bar from Foo cimport Bar in a .pyx file, is it safe to assume that the Bar we import at runtime is going to be coming from the same module whose .pxd we analyzed at compile-time? Is it at least safe to assume that the Bar we import at runtime will be the same *type* as the Bar we saw at compile-time? (The next obvious question: what other information can we assume is the same?) Of course, this sounds reasonable enough, but completely goes against the usual Python import semantics, since there are so many ways to change what will actually get imported at runtime. In general, it would be nice (specifically as our type inferencer becomes more sophisticated) to be able to provide Cython with as much information as possible about the module it's going to see at runtime -- not just what's in the .pxd. In fact, if you know at compile-time which module is going to be there at runtime, it would be helpful to be able to tell Cython here's the module, feel free to analyze it however you'd like. This makes sense even for import statements without a corresponding cimport. Would people be opposed to a @cython.staticimport decorator or the like? -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] semantics of import and cimport
You could disassemble the byte-code or parse the Python source and then do analysis, but I'm assuming you're not volunteering for that... :-) Not this week -- this is going to take a little bit longer. :) Do mean providing a set of annotations for pure Python code, so that we can read def f(x: cython.int) - cython.int in a pure Python file (using the Cython shadow module), and understand it? If so, +1. Yep, this is the obvious use case right now -- either simple functions (where we already can infer the type by analyzing the source code) or cases with explicit annotations. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Deferring closures to 0.14?
It's not the closures code that's holding anything up, as far as I know it's ready to go (with a small type inferencing issue, which could be easily resolved at first by disabling type inference for closures). It's the fact that Sage doesn't (didn't?) compile with the latest cython-devel tip. Yep, there's an issue with type inferencing and sage that took me a little while to track down. It's a case where I'm not sure what the right fix is -- it's a question of the semantics of import and cimport. I've got half an email to this list drafted about it, which I've finally got time to sit down and finish up right now ... Honestly, the much bigger holdup is that my house has apparently become a bastion of illness -- today is shaping up to be the second day this month that my daughter doesn't have a cold. :) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] new C++ syntax error
I am trying the new C++ syntax. I got the following error (all files attached): $ cython _hermes_common.pyx ... if len(self.attributes) != 0: TypeError: object of type 'NoneType' has no len() Yep, that's a bug. I'm pretty sure you just want to change that line in Cython/Compiler/Nodes.py from if len(self.attributes) != 0: to just if self.attributes: and you'll be good. Amusingly, I tried to grab the source files you attached and test this -- and hit a different bug (a problem in parsing). I don't know why -- you just posted and confirmed that we're using the same version (before I even asked!). Is there any chance your source files are slightly out of sync with the ones you posted? (Maybe forgot to save changes in your editor before posting?) Has anyone else tried and hit the same bug, or is my setup wonky? So I finally got a chance to sit down and look at this today. The first bug was easy, and luckily Robert Bradshaw was sitting right next to me for the second -- he immediately called the function where the error was going to be, and it was just a question of spotting it. I'm running tests right now, and I'll push as soon as those are done. I also had to make a few changes to your .pyx and .pxd files -- mostly little things. I'm attaching the versions I used, which now compile just fine. (I didn't actually test them at all, not having any of the code they're written against, just checked that they seem to be producing sensible .cpp files.) Definitely let us know if/when you run into more trouble -- a lot of this new C++ code just hasn't been run through its paces, and it's good to work the little bugs out. -cc _hermes_common.pxd Description: Binary data _hermes_common.pyx Description: Binary data ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] new C++ syntax error
Did you run python setup.py build_ext -i locally? The compiled Cython parser is not automatically rebuilt on source changes. Try make clean. Oh, it was just PEBKAC -- I had already made the fix I suggested to Ondrej in my local copy of Cython before I even downloaded the example. :) Ondrej -- even with that fix, you hit some other issues. I'm not going to look at this now -- hopefully tomorrow night or Friday, if no one beats me to it. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] getting rid of CPython's structmember.h
Hi Lisandro, BTW, Any suggestions about docstring generation? I mean: So I'm going to jump in and make a few comments on this thread without actually reading the whole thing first (always a dangerous move) ... I just want to give you the heads-up that support for docstrings on Cython properties is currently in iffy shape. I ended up wrestling with this a little bit while trying to get Cython to build Sage -- I'm planning on filing a handful of trac tickets (and hopefully working up some fixes) as soon as I've got an 0.13 alpha that can build Sage with type inferencing turned on. I think that no one's really tried out docstrings with properties before: they don't really play together well. Here are two big issues I've hit so far ... let's say we've got a class Foo with property my_prop: * I don't know how to get my hands on the docstring for Foo.my_prop.__get__ or Foo.my_prop.__set__ at all. I made a cursory glance through the C code, and wasn't sure that it was correctly getting attached anywhere, but it does at least appear in the C file. Actually, that's not completely true -- if the autotestdict is turned on, then the docstring comes up in mod.__test__. This was actually broken until a day or two ago (fixing it is how I ran into this whole mess in the first place). But it clearly should get attached to Foo somewhere. The fact that I couldn't find it another way is also why the test that I submitted goes via the __test__ dict's keys, instead of doing something more sane. (I plan to clean up the test once I fix the issues with properties.) * If you try to give a docstring to Foo.my_prop.__del__, Cython crashes. The underlying issue is this: we currently wrap the __set__ and __del__ methods into the setter part of a PyGetSetDef, so it's not clear where to attach the __del__ docstring -- or, at least, where to attach *both* a __set__ and __del__ docstring. (I don't think this is insurmountable, but we might need to think harder.) I was planning on starting a new thread once I had some more concrete suggestions for fixes. For comparison, Python attaches the docstrings to the Foo.my_prop.fget, fset, and fdel methods. (The Foo.my_prop.__get__ and whatnot all have generic docstrings.) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] new C++ syntax error
Hi Ondrej, I am trying the new C++ syntax. I got the following error (all files attached): $ cython _hermes_common.pyx ... if len(self.attributes) != 0: TypeError: object of type 'NoneType' has no len() Yep, that's a bug. I'm pretty sure you just want to change that line in Cython/Compiler/Nodes.py from if len(self.attributes) != 0: to just if self.attributes: and you'll be good. Amusingly, I tried to grab the source files you attached and test this -- and hit a different bug (a problem in parsing). I don't know why -- you just posted and confirmed that we're using the same version (before I even asked!). Is there any chance your source files are slightly out of sync with the ones you posted? (Maybe forgot to save changes in your editor before posting?) Has anyone else tried and hit the same bug, or is my setup wonky? In any event, there are definitely some bugs -- I'm currently trying to get type inference and the Sage library to get along, but I'll look at this soon if no one beats me to it. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] C++ method overloading?
The second error I got says: Invalid conversion from 'Python object' to 'double', which is something that should *always* work. I get the same error for other numeric types in other tests. Looks like Python object conversion is broken in cython-devel. I ran into this same problem earlier, and posted a potential fix on #506. It definitely solved the problem I was running into -- I'd be curious as to whether or not it solves the issues you're hitting ... -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
[Cython] Cython 0.13 and cython-closures branch
Hi all, So this has already been discussed in several other threads, but I think the current plan is: once Robert releases 0.12.1, we'll pull all the changes from there into the cython-closures branch, and turn cython-closures into cython-devel for 0.13. I was chatting with Robert, and I think I'm happy to be the release manager for 0.13, unless there are any objections ... A few quick notes: - I just finished (taking forever to) add a big battery of tests in the cython-closures branch. I ended up adding a wee bit of code, and then working out a few reference counting bugs that we hit with closures. I've committed all the tests into the closures branch (as a series of 5 files; gcc and g++ were choking with everything in one file). The tests are actually an old battery of tests in Scheme from the compilers courses I took in undergrad -- I wrote a little compiler that converted these to Python (modulo one issue I punted on due to a difference with the way Scheme handles scoping for letrec). I verified that Scheme, Python, and Cython gave the same results for each of these, and the refnanny didn't complain at all, so I feel pretty confident in the closures code. (Big thanks to the professors who first came up with the battery of tests, and yes, I did check with them before committing the tests.) - In the process of doing this, the refnanny complained a few times -- so I wrote myself some notes on how reference counting should be working. I realized these might be useful in general, so I've added a little extra text and posted it on the wiki: http://wiki.cython.org/refcounting - There is one place where I really didn't test out the closures code much, especially with an eye towards memory leaks/reference counting troubles, and that's error handling. If you have some code that uses nested functions and a bunch of error handling, I'd love to either (1) hear what happens when you run it with the refnanny turned on, or (2) have you send me a copy/post it somewhere so we can give it a go. This is good -- I think the next step is to start working on the generators implementation that came up in some other threads, and is outlined on the wiki. I'm planning on starting on that soon, if no one beats me to it -- and if my six-month old daughter decides to start sleeping through the night, I might actually move faster than molasses in January. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] [patch] Optional setuptools-based cython build
for ease of migration, distribute also puts (among other things) a 'setuptools' entry in sys.modules. so nothing else to do on the cython side. Will this still be true when the Distribute 0.7 series gets released? -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Bug day?
Thursday. Technically Thursday for all of us (though I'll be staying up for it). I think I'm going to try and do the same -- though I may start a little earlier (9PM-ish on Weds Seattle time). I'd say that the timing is bad for the US developers, but given all the folks I know in the US who might want to be involved, maybe the timing is in fact really good ... ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] -unstable and -devel branch
IMO cython-closures should be our new cython-unstable, to make sure it is tested and merged and keep the number of (cooperative) branches down. I don't mind either way. I agree that cython-closures can really use some more testing and shouldn't delay 0.12. I actually just posted a patch (actually, two patches, because I forgot to run tests before I pushed) to that branch. This is the bug that Robert ran into during our SD15 talk ... it should work now. I'm going to try and run a bunch more tests tonight, but I'm also +1 on cython-closures becoming the new cython-unstable. -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
Re: [Cython] Call for hosting of users' mailing list
Google Groups appears to be out of the question for Cython due to having to sign an EULA before getting posting access. I'm pretty sure one can post to the list without signing an EULA--at least once the moderators approve you. I'm pretty sure one needs an account to administer though. I don't find it too draconian, but for something like this finding the lowest common denominator is important. I'm just curious -- what was it about the Google Groups EULA that people disliked? I'm not trying to stir up trouble, but I don't know what the reasons were when setting up this list. ;) -cc ___ Cython-dev mailing list Cython-dev@codespeak.net http://codespeak.net/mailman/listinfo/cython-dev
