[Python-Dev] Re: Re: Re: Re: Patch review: [ 1094542 ] add Bunch type to collections module
Steven Bethard wrote: That sounds reasonable to me. I'll fix update to be a staticmethod. If people want other methods, I'll make sure they're staticmethods too.[1] Steve [1] In all the cases I can think of, staticmethod is sufficient -- the methods don't need to access any attributes of the Bunch class. If anyone has a good reason to make them classmethods instead, let me know... Great. I probably meant staticmethod. I don't use either much, so I don't really know the difference in the terminology. For a long time I stuck to 2.1 features for ipython and my other codes, and I seem to recall those appeared in 2.2. But you got what I meant :) Cheers, f ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: [PyCon] Reg: Registration
Aahz, writing as [EMAIL PROTECTED], wrote: It's still January 28 here -- register now! I don't know if we'll be able to extend the registration price beyond that. Just in case anybody else might be wondering when the early bird registration deadline is, I've asked the registration team to allow the early bird price as long as it's January 28th somewhere in the world. There have been rumors that Guido will not be attending PyCon this year. I am happy to scotch them by pointing out that Guido van Rossum's keynote address will be on its traditional Thursday morning. I look forward to joining you all to hear Guido speak on The State of Python. regards Steve -- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/ Holden Web LLC +1 703 861 4237 +1 800 494 3119 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Getting rid of unbound methods: patch available
Guido van Rossum wrote: Here's a patch that gets rid of unbound methods, as discussed here before. A function's __get__ method now returns the function unchanged when called without an instance, instead of returning an unbound method object. I thought the main reason for existence of unbound methods (for user-defined classes, at least) was so that if you screw up a super call by forgetting to pass self, or passing the wrong type of object, you get a more helpful error message. I remember a discussion about this some years ago, in which you seemed to think the ability to produce this message was important enough to justify the existence of unbound methods, even though it meant you couldn't easily have static methods (this was long before staticmethod() was created). Have you changed your mind about that? Also, surely unbound methods will still have to exist for C methods? Otherwise there will be nothing to ensure that C code is getting the object type it expects for self. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Getting rid of unbound methods: patch available
[Guido] Here's a patch that gets rid of unbound methods, as discussed here before. A function's __get__ method now returns the function unchanged when called without an instance, instead of returning an unbound method object. [Greg] I thought the main reason for existence of unbound methods (for user-defined classes, at least) was so that if you screw up a super call by forgetting to pass self, or passing the wrong type of object, you get a more helpful error message. Yes, Tim reminded me of this too. But he said he could live without it. :-) I remember a discussion about this some years ago, in which you seemed to think the ability to produce this message was important enough to justify the existence of unbound methods, even though it meant you couldn't easily have static methods (this was long before staticmethod() was created). Have you changed your mind about that? After all those years, I think the added complexity of unbound methods doesn't warrant having the convenience of the error message. Also, surely unbound methods will still have to exist for C methods? Otherwise there will be nothing to ensure that C code is getting the object type it expects for self. No, C methods have their own object type for that (which is logically equivalent to an unbound method). But there was a use case for unbound methods having to do with C methods of classic classes, in the implementation of built-in exceptions. Anyway, it's all moot because I withdrew the patch, due to the large amount of code that would break due to the missing im_class attribute -- all fixable, but enough not to want to break it all when 2.5 comes out. So I'm salting the idea up for 3.0. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's get rid of unbound methods
Tim Peters wrote: I expect that's because he stopped working on Zope code, so actually thinks it's odd again to see a gazillion methods like: class Registerer(my_base): def register(*args, **kws): my_base.register(*args, **kws) I second that! My PyGUI code is *full* of __init__ methods like that, because of my convention for supplying initial values of properties as keyword arguments. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Interpreter Thread Safety?
Evan Jones wrote: Sorry, I really meant *parallel* execution of Python code: Multiple threads simultaneously executing a Python program, potentially on different CPUs. There cannot be parallel threads on a single CPU - for threads to be truly parallel, you *must* have two CPUs, at a minimum. Python threads can run truly parallel, as long as one of them invoke BEGIN_ALLOW_THREADS. What I was trying to ask with my last email was what are the trouble areas? There are probably many that I am unaware of, due to my unfamiliarity the Python internals. I think nobody really remembers - ask Google for Python free threading. Greg Stein did the patch, and the main problem apparently was that the performance became unacceptable - apparently primarily because of dictionary locking. Right, but as said in a previous post, I'm not convinced that the current implementation is completely correct anyway. Why do you think so? (I see in your previous post that you claim it is not completely correct, but I don't see any proof). Wouldn't it be up to the programmer to ensure that accesses to shared objects, like containers, are serialized? In a truly parallel Python, two arbitrary threads could access the same container, and it would still work. If some containers cannot be used simultaneously in multiple threads, this might ask for a desaster. For example, with Java's collections, there are both synchronized and unsynchronized versions. I don't think this approach can apply to Python. Python users are used to completely thread-safe containers, and lots of programs would break if the container would suddenly throw exceptions. Furthermore, the question is what kind of failure you'ld expect if an unsynchronized dictionary is used from multiple threads. Apparently, Java guarantees something (e.g. that the interpreter won't crash) but even this guarantee would be difficult to make. For example, for lists, the C API allows direct access to the pointers in the list. If the elements of the list could change in-between, an object in the list might go away after you got the pointer, but before you had a chance to INCREF it. This would cause a crash shortly afterwards. Even if that was changed to always return a new refence, lots of code would break, as it would create large memory leaks (code would have needed to decref the list items, but currently doesn't - nor is it currently necessary). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
RE: [Python-Dev] Python Interpreter Thread Safety?
Martin v. Löwis writes: Due to some unfortunate historical reasons, there is code which enters free() without holding the GIL - and that is what the allocator specifically deals with. Except for this single case, all callers of the allocator are required to hold the GIL. Donovan Baarda writes: Just curious; is that one case a bug that needs fixing, or is the some reason this case can't be changed to use the GIL? Surely making it mandatory for all free() calls to hold the GIL is easier than making the allocator deal with the one case where this isn't done. What Martin is trying to say here is that it _IS_ mandatory to hold the GIL when calling free(). However, there is some very old code in existance (written by other people) which calls free() without holding the GIL. We work very hard to provide backward compatibility, so we are jumping through hoops to ensure that even this old code which is violating the rules doesn't get broken. -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Interpreter Thread Safety?
... [Evan Jones] What I was trying to ask with my last email was what are the trouble areas? There are probably many that I am unaware of, due to my unfamiliarity the Python internals. Google on Python free threading. That's not meant to be curt, it's just meant to recognize that the task is daunting and has been discussed often before. [Martin v. Löwis] Due to some unfortunate historical reasons, there is code which enters free() without holding the GIL - and that is what the allocator specifically deals with. Right, but as said in a previous post, I'm not convinced that the current implementation is completely correct anyway. Sorry, I haven't had time for this. From your earlier post: For example, is it possible to call PyMem_Free from two threads simultaneously? Possible but not legal; undefined behavior if you try. See the Thread State and the Global Interpreter Lock section of the Python C API manual. ... only the thread that has acquired the global interpreter lock may operate on Python objects or call Python/C API functions There are only a handful of exceptions to the last part of that rule, concerned with interpreter and thread startup and shutdown, and they're explicitly listed in that section. The memory-management functions aren't among them. In addition, it's not legal to call PyMem_Free regardless unless the pointer passed to it was originally obtained from another function in the PyMem_* family (that specifically excludes memory obtained from a PyObject_* function). In a release build, all of the PyMem_* allocators resolve directly to the platform malloc or realloc, and all PyMem_Free has to determine is that they *were* so allocated and thus call the platform free() directly (which is presumably safe to call without holding the GIL). The hacks in PyObject_Free (== PyMem_Free) are there solely so that question can be answered correctly in the absence of holding the GIL. That question == does pymalloc control the pointer passed to me, or does the system malloc?. In return, that hack is there solely because in much earlier versions of Python extension writers got into the horrible habit of allocating object memory with PyObject_New but releasing it with PyMem_Free, and because indeed Python didn't *have* a PyObject_Free function then. Other extension writers were just nuts, mixing PyMem_* calls with direct calls to system free/malloc/realloc, and ignoring GIL issues for all of those. When pymalloc was new, we went to insane lengths to avoid breaking that stuff, but enough is enough. Since the problem is that threads could call PyMem_Free without holding the GIL, it seems to be that it is possible. Yes, but not specific to PyMem_Free. It's clearly _possible_ to call _any_ function from multiple threads without holding the GIL. Shouldn't it also be supported? No. If what they want is the system malloc/realloc/free, that's what they should call. In the current memory allocator, I believe that situation can lead to inconsistent state. Certainly, but only if pymalloc controls the memory blocks. If they were actually obtained from the system malloc, the only part of pymalloc that has to work correctly is the Py_ADDRESS_IN_RANGE() macro. When that returns false, the only other thing PyObject_Free() does is call the system free() immediately, then return. None of pymalloc's data structures are involved, apart from the hacks ensuring that the arena of base addresses is safe to access despite potentlly current mutation-by-appending. ... Basically, if a concurrent memory allocator is the requirement, It isn't. The attempt to _exploit_ the GIL by doing no internal locking of its own is 100% deliberate in pymalloc -- it's a significant speed win (albeit on some platforms more than others). then I think some other approach is necessary. If it became necessary, that's what this section of obmalloc is for: SIMPLELOCK_DECL(_malloc_lock) #define LOCK() SIMPLELOCK_LOCK(_malloc_lock) #define UNLOCK()SIMPLELOCK_UNLOCK(_malloc_lock) #define LOCK_INIT() SIMPLELOCK_INIT(_malloc_lock) #define LOCK_FINI() SIMPLELOCK_FINI(_malloc_lock) You'll see that PyObject_Free() calls LOCK() and UNLOCK() at appropriate places already, but they have empty expansions now. Back to the present: [Martin] Again, the interpreter supports multi-threading today. Removing the GIL is more difficult, though - nearly any container object (list, dictionary, etc) would have to change, plus the reference counting (which would have to grow atomic increment/decrement). [Evan] Wouldn't it be up to the programmer to ensure that accesses to shared objects, like containers, are serialized? For example, with Java's collections, there are both synchronized and unsynchronized versions. Enormous mounds of existing threaded Python code freely manipulates lists and dicts without explicit locking now. We can't break that -- and wouldn't want to. Writing
Re: [Python-Dev] Python Interpreter Thread Safety?
On Jan 28, 2005, at 19:44, Martin v. Löwis wrote: Python threads can run truly parallel, as long as one of them invoke BEGIN_ALLOW_THREADS. Except that they are really executing C code, not Python code. I think nobody really remembers - ask Google for Python free threading. Greg Stein did the patch, and the main problem apparently was that the performance became unacceptable - apparently primarily because of dictionary locking. Thanks, I found the threads discussing it. Right, but as said in a previous post, I'm not convinced that the current implementation is completely correct anyway. Why do you think so? (I see in your previous post that you claim it is not completely correct, but I don't see any proof). There are a number of issues actually, but as Tim points, only if the blocks are managed by PyMalloc. I had written a description of three of them here, but they are not relevant. If the issue is calling PyMem_Free with a pointer that was allocated with malloc() while PyMalloc is doing other stuff, then no problem: That is possible to support, but I'll have to think rather hard about some of the issues. For example, for lists, the C API allows direct access to the pointers in the list. If the elements of the list could change in-between, an object in the list might go away after you got the pointer, but before you had a chance to INCREF it. This would cause a crash shortly afterwards. Even if that was changed to always return a new refence, lots of code would break, as it would create large memory leaks (code would have needed to decref the list items, but currently doesn't - nor is it currently necessary). Ah! Right. In Java, the collections are all actually written in Java, and run on the VM. Thus, when some concurrent weirdness happens, it just corrupts the application, not the VM. However, in Python, this could actually corrupt the interpreter itself, crashing the entire thing with a very ungraceful Segmentation Fault or something similar. Evan Jones ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com