[Python-Dev] yappi - any thoughts?
Hi there, yappi(Yet Another Python Profiler) is a multithreaded profiler for 2.x series(do not know if it will work on 3k, not tested yet). I have done my best to make it efficient as possible.(details are under the technical bullet in the website). It is aimed for long-running programs, to attach/detach profiler and retrieve stats on the fly. The current cProfile module needs substantial amount of work to accomplish this task and also have some problems with recursive functions, basically that is why I have written a profiler from scratch. yappi is currently(v0.2) modifying the profilefunc variable fo the ThreadState object to profile an exsiting thread. The newly created threads are catched via threading.setprofile() call(from VM to our extension). And for the deleted threads, we rely on the recycling of the ThreadState objects.(Please see the website for details.) I have tested it under a game server (100k players per-day) for about a month for any issues and now releasing it. Please see: http://code.google.com/p/yappi/ Any thoughts/comments/ideas, anything?:) -- Sumer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it
Am Freitag, 30. Oktober 2009 03:58:16 schrieb Steven D'Aprano: To clarify point 3, given: x = set.get() y = set.get() then x and y will only be the same element if set has length one. However, given: x = set.get() set.add(el) set.remove(el) y = set.get() there are no guarantees about x and y being different. I believe that the patch supplied by Willi Richart implemented these behaviours. http://bugs.python.org/issue7212 Actually, no. The patch makes no assumption about x and y being different. It does not try to extend the core functionality of set nor does it need to maintain any state that would be necessary for that. It is just a more obvious and cleaner way for saying for x in some_set: break. So, as Raymond says, it is the Pythonic version of arb in setl. wr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
Jesse Noller wrote: On Thu, Oct 29, 2009 at 8:31 PM, R. David Murray rdmur...@bitdance.com wrote: I'd say that particular one is a bug in the tests. If /dev/shm is not available and is required, then the tests should be skipped with an appropriate message. It would also secondarily be an issue with the buildbot fleet, since multiprocessing would then not be getting thoroughly tested by those buildbots. Fwiw: The tests skip on those platforms; but multiprocessing can't function properly on platforms like that. I'm confused: first you said they fail, now you say they get skipped. Which one is it? I agree with R. David's analysis: if they fail, it's a multiprocessing bug, if they get skipped, it's a flaw in the build slave configuration (but perhaps only slightly so, because it is good if both cases are tested - and we do have machines also that provide /dev/shm). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
But the real reason for having a buildbot category (or at least a keyword) would be to be able to tag all bugs that are currently making buildbots fail that are _not_ the result of a recent checkin. This would make the task of finding the bugs that need to be cleaned up to stabilize the buildbot fleet easier. I'm currently aware of issues 4970, 3892, and 6462 in this category, and there are a few more that we can/will file if we continue to pay attention to the failure reports now arriving on the irc channel. That's convincing; I've created a buildbot keyword. I gave it the description indicates that tests fail on a buildslave for uncertain reasons If that is indeed the intended purpose of this classification, please keep it in mind when assigning the tag. If I misunderstood the purpose of the keyword, please provide a better description. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssl module
Is there a place where the status of the ssl module is summarized The documentation of the ssl module should describe its features correctly and precisely. or a better place to discuss this? I could try to provide contributions or further details if appropriate. For contributions, this is indeed the right place for discussion. For mere inquiries for help, other places are better (such as python-list). For specific feature requests, use the bug tracker. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yappi - any thoughts?
Any thoughts/comments/ideas, anything?:) Announce it to comp.lang.python.announce if you haven't done so, yet. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
On Fri, Oct 30, 2009 at 4:53 AM, Martin v. Löwis mar...@v.loewis.de wrote: I'm confused: first you said they fail, now you say they get skipped. Which one is it? I agree with R. David's analysis: if they fail, it's a multiprocessing bug, if they get skipped, it's a flaw in the build slave configuration (but perhaps only slightly so, because it is good if both cases are tested - and we do have machines also that provide /dev/shm). They failed until we had the tests skip those platforms - at the time, I felt that it was more of a bug with the build slave configuration than a multiprocessing issue, I don't like skipping tests unless the platform fundamentally isn't supported (e.g. broken semaphores for some actions on OS/X) - linux platforms support this functionality just fine - except when in locked-down chroot jails. The only reason I brought it up was to point out the a buildbot configuration on a given host can make tests fail even if those tests would normally pass on that operating system. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
On 12:55 pm, jnol...@gmail.com wrote: On Fri, Oct 30, 2009 at 4:53 AM, Martin v. Löwis mar...@v.loewis.de wrote: I'm confused: first you said they fail, now you say they get skipped. Which one is it? I agree with R. David's analysis: if they fail, it's a multiprocessing bug, if they get skipped, it's a flaw in the build slave configuration (but perhaps only slightly so, because it is good if both cases are tested - and we do have machines also that provide /dev/shm). They failed until we had the tests skip those platforms - at the time, I felt that it was more of a bug with the build slave configuration than a multiprocessing issue, I don't like skipping tests unless the platform fundamentally isn't supported (e.g. broken semaphores for some actions on OS/X) - linux platforms support this functionality just fine - except when in locked-down chroot jails. The only reason I brought it up was to point out the a buildbot configuration on a given host can make tests fail even if those tests would normally pass on that operating system. Just as a build slave can be run in a chroot, so can any other Python program. This is a real shortcoming of the multiprocessing module. It's entirely possible that people will want to run Python software in chroots sometimes. So it's proper to acknowledge that this is an unsupported environment. The fact that the kernel in use is the same as the kernel in use on another supported platform is sort of irrelevant. The kernel is just one piece of the system, there are many other important pieces. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
On Fri, 30 Oct 2009 at 08:55, Jesse Noller wrote: On Fri, Oct 30, 2009 at 4:53 AM, Martin v. L?wis mar...@v.loewis.de wrote: I'm confused: first you said they fail, now you say they get skipped. Which one is it? I agree with R. David's analysis: if they fail, it's a multiprocessing bug, if they get skipped, it's a flaw in the build slave configuration (but perhaps only slightly so, because it is good if both cases are tested - and we do have machines also that provide /dev/shm). They failed until we had the tests skip those platforms - at the time, I felt that it was more of a bug with the build slave configuration than a multiprocessing issue, I don't like skipping tests unless the platform fundamentally isn't supported (e.g. broken semaphores for some actions on OS/X) - linux platforms support this functionality just fine - except when in locked-down chroot jails. As Martin pointed out, Python supports both configurations (chroot and non-chroot), and needs to be tested in both. Somewhere we should probably have a list of what tests are getting skipped on what buildslaves so we can inspect the buildbot fleet for complete coverage, but I'm not sure who is going to volunteer to create and maintain that list :) The only reason I brought it up was to point out the a buildbot configuration on a given host can make tests fail even if those tests would normally pass on that operating system. Yes, and that's a kind of ticket that should end up getting tagged with the new 'buildbot' keyword (thanks, Martin), IMO. --David___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
On Fri, Oct 30, 2009 at 10:15 AM, exar...@twistedmatrix.com wrote: On 12:55 pm, jnol...@gmail.com wrote: On Fri, Oct 30, 2009 at 4:53 AM, Martin v. Löwis mar...@v.loewis.de wrote: I'm confused: first you said they fail, now you say they get skipped. Which one is it? I agree with R. David's analysis: if they fail, it's a multiprocessing bug, if they get skipped, it's a flaw in the build slave configuration (but perhaps only slightly so, because it is good if both cases are tested - and we do have machines also that provide /dev/shm). They failed until we had the tests skip those platforms - at the time, I felt that it was more of a bug with the build slave configuration than a multiprocessing issue, I don't like skipping tests unless the platform fundamentally isn't supported (e.g. broken semaphores for some actions on OS/X) - linux platforms support this functionality just fine - except when in locked-down chroot jails. The only reason I brought it up was to point out the a buildbot configuration on a given host can make tests fail even if those tests would normally pass on that operating system. Just as a build slave can be run in a chroot, so can any other Python program. This is a real shortcoming of the multiprocessing module. It's entirely possible that people will want to run Python software in chroots sometimes. So it's proper to acknowledge that this is an unsupported environment. The fact that the kernel in use is the same as the kernel in use on another supported platform is sort of irrelevant. The kernel is just one piece of the system, there are many other important pieces. Jean-Paul I'm well aware of that. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it
On Fri, 30 Oct 2009 03:00:27 pm Raymond Hettinger wrote: [skipping to the last paragraph] Sorry for so many questions Don't be sorry. These are good questions, and I'll try to answer them. But keep in mind that this isn't my proposal -- I vary between +0 and +1 on the proposal depending on the time of the day *wink* (1) it will only fail if the set is empty; Just like next() except that next() gives you the option to supply a default and can be used on any iterator (perhaps iter(s) or itertools.cycle(s) etc). Yes. I had considered suggesting that sets become their own iterator, so you could do the following: s = set([1,2]) next(s) 2 but that leads to the problem that if you accept a set from somewhere else, you have no idea whether next(s) will succeed or raise StopIteration. It may have already been exhausted before it reached you. (2) it should be efficient; Is this about optimization? Not primarily. Perhaps I should have said it should not be inefficient. It's primarily about there being One Obvious Way to extract an arbitrary item from a set -- this is a reoccurring question on comp.lang.python. Being efficient is a bonus, but it shouldn't be inefficient. I wouldn't expect x=s.get() to beat for x in s: break. Attribute lookup and method calls usually are slower than equivalents using built-in syntax with specific opcodes. Obviously any hypothetical solution would need to be benchmarked, but I wouldn't be concerned if s.get() was marginally slower than for `x in s: break`. When people are left to come up with their own ad hoc solutions, we've seen some poor solutions. I've seen, possibly in this very thread, the following O(N) suggestions: for x in s: pass x = list(s)[0] The usual technique people tend to come up with is: x = s.pop() s.add(x) Which strikes many people (including myself) as inelegant. Surely get an element is a more fundamental operation than get an element and remove it? (3) if you call it repeatedly on a set without modifying the set, you will cycle through each element in turn in some unspecified arbitrary order. What's wrong with using next()? That is what it's for. You can't call next() on a set. You have to call next(iter(set)). From Python 3.0: next(set([1,2])) Traceback (most recent call last): File stdin, line 1, in module TypeError: set object is not an iterator next(iter(set([1,2]))) 1 Unless I missed something, this is so unobvious that nobody has suggested it in the thread yet. What about this proposal is specific to sets, i.e. why don't you want the same thing for lists. tuples, strings, file objects, or any other iterable? Sequences such as lists, tuples and strings have easy random access. It's easy to get an arbitrary element: pick an index i by whatever method you like, and get seq[i]. Many file objects are similar, you have random access with seek(). It is unpractical and an over-generalisation to apply this to general iterables, for reasons I'm sure I don't need to go into. But sets and frozensets aren't lazily generated streams, they actually do store the elements inside their data structure in the same way that lists do. Does this proposal pass the test of being self-descriptive? Can you write a code fragment that exercises the cycling behavior, show it to another programmer, and have them correctly deduce what the code does (i.e. that different values are returned, that it fails when the set it empty, that it wraps around and never terminates)? Can they readily differentiate it for dict.get() which has decidedly different semantics? I don't expect ConfigParser.get() to have the same semantics as dict.get(), and they're much more closely related than sets and dicts. I don't understand why so many people apparently have a problem with the name get(), but that's okay, I'm not wedded to the idea either. If folks prefer a different name, by all means suggest it. I like the name suggested by Wikipedia, pick. As for being self-descriptive, I don't know, I haven't tried the experiment. To clarify point 3, given: x = set.get() y = set.get() then x and y will only be the same element if set has length one. So, it can't even be used for looping through a set because there is no termination? That's a feature, not a bug! This is not intended to be a replacement for standard set iteration. Anyone writing the following: for _ in len(s): process(s.get()) instead of for x in s: process(x) will be taken out and slapped with a large fish. My reasoning for the given behaviour is as follows: * If you want the same element twice from a set, the One Obvious Way is to get an element from the set (somehow!) and assign it to a local variable: x = s.get() process(x) # later... process(x) So having get() return the same value each time is not useful or necessary. * If you want a random element on each call, the One Obvious Way is
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
Steven D'Aprano steve at pearwood.info writes:
If you can think of any other way to efficiently cycle over the elements
in a set, I'm all for it :)
How about for x in s?
Or if you want to cycle:
s = set('abc')
it = itertools.cycle(s)
next(it)
'a'
next(it)
'c'
next(it)
'b'
next(it)
'a'
Or if you don't want the overhead of itertools.cycle() keeping a copy of the
set's elements:
s = set('abc')
it = itertools.chain.from_iterable(itertools.cycle([s]))
next(it)
'a'
next(it)
'c'
next(it)
'b'
next(it)
'a'
next(it)
'c'
next(it)
'b'
I can't say I've seen one in any other languages, but Wikipedia
lists pick as a fundamental set operation:
pick(S): returns an arbitrary element of S.
Well, it's an arbitrary element. It isn't specified that it will try to return
different results in a row to satisfy the developer's aesthetical preferences...
This page claims that Icon has an operator that returns a random element
of a set:
? set( [1, 2, 3, 4, 5] )
random != arbitrary != weak-guaranteedly distinct
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it
Steven D'Aprano wrote: So you want to introduce additional, hidden state to sets? (to make sure that successive invocations return different values) If you can think of any other way to efficiently cycle over the elements in a set, I'm all for it :) for x in itertools.cycle(s): # this is an infinite loop Having a pick() or get() method that returns an arbitrary member of a set makes sense to me. Having any state on the set that guarantees successive calls to get will return different values feels wrong - creating an object with that extra state is what iter(s) is for. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
Hello, Sorry for the little redundancy, I would like to underline Jean-Paul's suggestion here: Le Sun, 25 Oct 2009 14:05:12 +, exarkun a écrit : I think that money can help in two ways in this case. First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. Amazon's EC2 service is supported out of the box in the latest release. I'm not a PSF member, but it seems to me that the PSF could ask Amazon (or any other virtual machine business anyway) to donate a small number of permanent EC2 instances in order to run buildslaves on. After all, big companies often like sponsoring open-source projects, especially when the project is well-known and the donation is cheap for them. This would have several advantages: - the machines are administered by the provider: we don't have to worry about failed hardware, connectivity loss etc. - any Python core developer could get ssh access to the VMs to run tests directly, since they would be dedicated buildbot instances - they are not tied to a particular owner when it comes to fixing system problems, which means we eliminate a single point of failure: if a volunteer gets demotivated/bored/missing in action, someone can replace him/her easily - there are a number of various OS images available (of course, we still need competent people to install the required software -- buildbot, etc.) Since I've never used any such service (cloud-based VMs), I'm not sure what the downsides would be. But it seems to be that it would be at least worth trying. Right now we have around 15 buildbots but two thirds of them are down, the others sometimes fail or disconnect in erratic ways and it's difficult for regular core developers to be aware of what's precisely going on. Of course this could also be a broken idea, for whatever reason I'm not aware of. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Fri, Oct 30, 2009 at 04:21:06PM +, Antoine Pitrou wrote: Hello, Sorry for the little redundancy, I would like to underline Jean-Paul's suggestion here: Le Sun, 25 Oct 2009 14:05:12 +, exarkun a ??crit??: I think that money can help in two ways in this case. First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. Amazon's EC2 service is supported out of the box in the latest release. I'm not a PSF member, but it seems to me that the PSF could ask Amazon (or any other virtual machine business anyway) to donate a small number of permanent EC2 instances in order to run buildslaves on. [ ... ] I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux ia64; Linux x86; and Mac OS X. Others have made similar offers. The architectures supported by the cloud services don't really add anything (and generally don't have Mac OS X support, AFAIK). What we really need (IMO) is someone to dig into the tests to figure out which tests fail randomly and why, and to fix them on specific architectures that most of us don't personally use. This is hard work that is neither glamorous nor popular. I think the idea of paying a dedicated developer to make the CPython+buildbot tests reliable is better, although I would still be -0 on it (I don't think the PSF should be paying for this kind of thing at all). cheers, --titus -- C. Titus Brown, c...@msu.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
Le vendredi 30 octobre 2009 à 09:31 -0700, C. Titus Brown a écrit : [ ... ] I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux ia64; Linux x86; and Mac OS X. Others have made similar offers. The architectures supported by the cloud services don't really add anything (and generally don't have Mac OS X support, AFAIK). Well these VMs would have to run buildslaves on them, then. Are you ready to host some and connect them to the current buildbot infrastructure? (VMs without buildslaves are less interesting IMO) What we really need (IMO) is someone to dig into the tests to figure out which tests fail randomly and why, and to fix them on specific architectures that most of us don't personally use. This is hard work that is neither glamorous nor popular. I'm sure some of us are ready to do so (*). The situation has already improved quite a lot in the recent times. But fixing platform- or, worse, setup-specific issues often requires shell access to the target system, otherwise you spend too much time trying fixes on the SVN and waiting for the buildbot to react. (*) After all, if we weren't, we wouldn't even care about buildbots, we'd be content with running the test suite on our own machines I think the idea of paying a dedicated developer to make the CPython+buildbot tests reliable is better, although I would still be -0 on it (I don't think the PSF should be paying for this kind of thing at all). Paying developers in volunteer communities is always more contentious than paying for other kinds of resources. (It's generally more expensive too) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [TIP] Possible language summit topic: buildbots
On Sun, Oct 25, 2009 at 9:13 AM, exar...@twistedmatrix.com wrote: On 12:48 pm, c...@msu.edu wrote: [snip] The most *exciting* part of pony-build, apart from the always-riveting spectacle of titus rediscovering problems that buildbot solved 5 years ago, is the loose coupling of recording server to the build slaves and build reporters. My plan is to enable a simple and lightweight XML-RPC and/or REST-ish interface for querying the recording server from scripts or other Web sites. This has Brett aquiver with anticipation, I gather -- no more visual inspection of buildbot waterfall pages ;) BuildBot has an XML-RPC interface. So Brett can probably do what he wants with BuildBot right now. ... but pony-build follows a different model ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Fri, Oct 30, 2009 at 05:41:39PM +0100, Antoine Pitrou wrote: Le vendredi 30 octobre 2009 ?? 09:31 -0700, C. Titus Brown a ??crit : [ ... ] I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux ia64; Linux x86; and Mac OS X. Others have made similar offers. The architectures supported by the cloud services don't really add anything (and generally don't have Mac OS X support, AFAIK). Well these VMs would have to run buildslaves on them, then. Are you ready to host some and connect them to the current buildbot infrastructure? (VMs without buildslaves are less interesting IMO) No, I'm not willing to spend the time to install and maintain buildbot. But I'm happy to give the necessary access to those who are interested and willing. (...and let me tell you, getting these !#%!#$! Windows VMs up and running already took an immense amount of effort ;) What we really need (IMO) is someone to dig into the tests to figure out which tests fail randomly and why, and to fix them on specific architectures that most of us don't personally use. This is hard work that is neither glamorous nor popular. I'm sure some of us are ready to do so (*). The situation has already improved quite a lot in the recent times. But fixing platform- or, worse, setup-specific issues often requires shell access to the target system, otherwise you spend too much time trying fixes on the SVN and waiting for the buildbot to react. (*) After all, if we weren't, we wouldn't even care about buildbots, we'd be content with running the test suite on our own machines I look forward to it! cheers, --titus -- C. Titus Brown, c...@msu.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [TIP] Possible language summit topic: buildbots
On Fri, Oct 30, 2009 at 11:42:30AM -0500, Olemis Lang wrote: On Sun, Oct 25, 2009 at 9:13 AM, exar...@twistedmatrix.com wrote: On 12:48 pm, c...@msu.edu wrote: [snip] The most *exciting* part of pony-build, apart from the always-riveting spectacle of titus rediscovering problems that buildbot solved 5 years ago, is the loose coupling of recording server to the build slaves and build reporters. ?My plan is to enable a simple and lightweight XML-RPC and/or REST-ish interface for querying the recording server from scripts or other Web sites. ?This has Brett aquiver with anticipation, I gather -- no more visual inspection of buildbot waterfall pages ;) BuildBot has an XML-RPC interface. ?So Brett can probably do what he wants with BuildBot right now. ... but pony-build follows a different model I'd rather not get into discussions of why my vaporware is going to be way, way better than anything anyone else could possibly do -- that's flamebait and not very friendly, in the end. Let's just say that I'm wasting my own time on it to scratch my own itch and leave it at that! thanks, --titus -- C. Titus Brown, c...@msu.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
2009/10/30 C. Titus Brown c...@msu.edu: On Fri, Oct 30, 2009 at 04:21:06PM +, Antoine Pitrou wrote: Hello, Sorry for the little redundancy, I would like to underline Jean-Paul's suggestion here: Le Sun, 25 Oct 2009 14:05:12 +, exarkun a ??crit??: I think that money can help in two ways in this case. First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. Amazon's EC2 service is supported out of the box in the latest release. I'm not a PSF member, but it seems to me that the PSF could ask Amazon (or any other virtual machine business anyway) to donate a small number of permanent EC2 instances in order to run buildslaves on. [ ... ] I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux ia64; Linux x86; and Mac OS X. Others have made similar offers. The architectures supported by the cloud services don't really add anything (and generally don't have Mac OS X support, AFAIK). As Antione pointed out, it's not clear (at least, it isn't to me) what that leaves to be done. As a counter-offer: Given remote access to however many Windows VMs you want to provide, I'll get them up and running with buildslaves on them. If that requires software such as Visual Studio, I have copies via the MSDN licenses that I am happy to provide. Once things are up and running, I'll be prepared to do basic care and feeding of the buildslave, but as my time is limited, it would be nice if others would pitch in to help. In other words, if it's setup effort that's lacking, I'll provide it. As long as someone else can cover systems admin, and we get some level of volunteers to cover ongoing support, that should give us better Windows coverage on the buildbots. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Fri, Oct 30, 2009 at 04:49:51PM +, Paul Moore wrote: 2009/10/30 C. Titus Brown c...@msu.edu: On Fri, Oct 30, 2009 at 04:21:06PM +, Antoine Pitrou wrote: Hello, Sorry for the little redundancy, I would like to underline Jean-Paul's suggestion here: Le Sun, 25 Oct 2009 14:05:12 +, exarkun a ??crit??: I think that money can help in two ways in this case. First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. ?BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. ?Amazon's EC2 service is supported out of the box in the latest release. I'm not a PSF member, but it seems to me that the PSF could ask Amazon (or any other virtual machine business anyway) to donate a small number of permanent EC2 instances in order to run buildslaves on. [ ... ] I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux ia64; Linux x86; and Mac OS X. ?Others have made similar offers. ?The architectures supported by the cloud services don't really add anything (and generally don't have Mac OS X support, AFAIK). As Antione pointed out, it's not clear (at least, it isn't to me) what that leaves to be done. Great! We've solved the problem ;) As a counter-offer: Given remote access to however many Windows VMs you want to provide, I'll get them up and running with buildslaves on them. If that requires software such as Visual Studio, I have copies via the MSDN licenses that I am happy to provide. I, too, have MSDN licenses, and I have functioning build environments on all of the VMs (I think -- I've only tested Win XP currently: http://lyorn.idyll.org/ctb/pb-dev/python/detail?result_key=8276 ) I also have an OS X 10.5 machine that I can let you into through a firewall; it's building Python 2.7 quite nicely: http://lyorn.idyll.org/ctb/pb-dev/python/detail?result_key=8229 Once things are up and running, I'll be prepared to do basic care and feeding of the buildslave, but as my time is limited, it would be nice if others would pitch in to help. I would be somewhat unhappy about giving more than three or four people admin access, but am prepared to lie back and think of England. --titus -- C. Titus Brown, c...@msu.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Cloud build slaves (was Re: Possible language summit topic: buildbots)
On 04:31 pm, c...@msu.edu wrote: On Fri, Oct 30, 2009 at 04:21:06PM +, Antoine Pitrou wrote: Hello, Sorry for the little redundancy, I would like to underline Jean-Paul's suggestion here: Le Sun, 25 Oct 2009 14:05:12 +, exarkun a ??crit??: I think that money can help in two ways in this case. First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. Amazon's EC2 service is supported out of the box in the latest release. I'm not a PSF member, but it seems to me that the PSF could ask Amazon (or any other virtual machine business anyway) to donate a small number of permanent EC2 instances in order to run buildslaves on. [ ... ] I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux ia64; Linux x86; and Mac OS X. Okay, let's move on this. Martin has, I believe, said that potential slave operators only need to contact him to get credentials for new slaves. Can you make sure to follow up with him to get slaves running on these machines? Or would you rather give out access to someone else and have them do the build slave setup? Others have made similar offers. I'll similarly encourage them to take action, then. Do you happen to remember who? The architectures supported by the cloud services don't really add anything (and generally don't have Mac OS X support, AFAIK). That's not entirely accurate. Currently, CPython has slaves on these platforms: - x86 - FreeBSD - Windows XP - Gentoo Linux - OS X - ia64 - Ubuntu Linux - Alpha - Debian Linux So, assuming we don't want to introduce any new OS, Amazon could fill in the following holes: - x86 - Ubuntu Linux - ia64 - FreeBSD - Windows XP - Gentoo Linux So very modestly, that's 4 currently missing slaves which Amazon's cloud service *does* add. It's easy to imagine further additions it could make as well. What we really need (IMO) is someone to dig into the tests to figure out which tests fail randomly and why, and to fix them on specific architectures that most of us don't personally use. This is hard work that is neither glamorous nor popular. Sure. That's certainly necessary. I don't think anyone is suggesting that it's not. Fortunately, adding more build slaves is not mutually exclusive with a developer fixing bugs in CPython. I think the idea of paying a dedicated developer to make the CPython+buildbot tests reliable is better, although I would still be -0 on it (I don't think the PSF should be paying for this kind of thing at all). I hope everyone is on board with the idea of fixing bugs in CPython, either in the actual implementation of features or in the tests for those features. That being the case, the discussion of whether or not the PSF should try to fund such a task is perhaps best discussed on the PSF members list. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [TIP] Possible language summit topic: buildbots
On 04:42 pm, ole...@gmail.com wrote: On Sun, Oct 25, 2009 at 9:13 AM, exar...@twistedmatrix.com wrote: On 12:48 pm, c...@msu.edu wrote: [snip] The most *exciting* part of pony-build, apart from the always- riveting spectacle of titus rediscovering problems that buildbot solved 5 years ago, is the loose coupling of recording server to the build slaves and build reporters. My plan is to enable a simple and lightweight XML-RPC and/or REST-ish interface for querying the recording server from scripts or other Web sites. �This has Brett aquiver with anticipation, I gather -- no more visual inspection of buildbot waterfall pages ;) BuildBot has an XML-RPC interface. So Brett can probably do what he wants with BuildBot right now. ... but pony-build follows a different model But BuildBot exists, is deployed, and can be used now, without waiting. (Sorry, I don't really understand what point you were hoping to make with your message, so I just thought I'd follow up in the same style and hope that you'll understand my message even if I don't understand yours :). Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (10/23/09 - 10/30/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2483 open (+31) / 16582 closed (+19) / 19065 total (+50) Open issues with patches: 989 Average duration of open issues: 675 days. Median duration of open issues: 430 days. Open Issues Breakdown open 2447 (+31) pending35 ( +0) Issues Created Or Reopened (53) ___ distutils and IronPython compatibility 10/27/09 CLOSED http://bugs.python.org/issue7071reopened michael.foord 26backport Popen blocks program from another thread 10/23/09 CLOSED http://bugs.python.org/issue7193reopened dgriff1 test_thread is flaky 10/23/09 CLOSED http://bugs.python.org/issue7194created pitrou Value error 'path is on drive c: start on drive d:' in os.path.r 10/23/09 http://bugs.python.org/issue7195created jaraco Clarify str.split() behavior 10/24/09 http://bugs.python.org/issue7196created gagenellina patch test_multiprocessing crashes under Windows when run in verbose m 10/24/09 http://bugs.python.org/issue7197created pitrou patch csv.writer 10/24/09 CLOSED http://bugs.python.org/issue7198created zacktu Doc: Logging level order seems inconsistent 10/24/09 CLOSED http://bugs.python.org/issue7199created tjreedy multiprocessing deadlock on Mac OS X when queue collected before 10/24/09 http://bugs.python.org/issue7200created bquinlan double Endian problem and more on arm10/24/09 http://bugs.python.org/issue7201created mancausoft python setup.py MYCOMMAND --verbose does not yield an unrecogn 10/25/09 http://bugs.python.org/issue7202created zooko fixer for map(None, ...) needs to consider multi-argument case 10/25/09 CLOSED http://bugs.python.org/issue7203created georg.brandl Strange shebang line in test_pep263 10/25/09 CLOSED http://bugs.python.org/issue7204created wiz bz2file deadlock 10/25/09 CLOSED http://bugs.python.org/issue7205created rbcollins patch 64 bit python fails on Windows 7 10/25/09 http://bugs.python.org/issue7206created richo test_telnetlib fails on OS X 10.610/26/09 CLOSED http://bugs.python.org/issue7207created mark.dickinson patch, needs review Getpass echo's password to screen on 2.6, but not on 2.5 or 3.1 10/26/09 http://bugs.python.org/issue7208created p...@fodder.org.uk patch Prevents uint_t from being used on QNX 10/26/09 http://bugs.python.org/issue7209created kraai patch Proposed Syntax Checks in Test Suite 10/26/09 http://bugs.python.org/issue7210created ChuckRhode patch
Re: [Python-Dev] [TIP] Possible language summit topic: buildbots
On Fri, Oct 30, 2009 at 11:45 AM, C. Titus Brown c...@msu.edu wrote: On Fri, Oct 30, 2009 at 11:42:30AM -0500, Olemis Lang wrote: On Sun, Oct 25, 2009 at 9:13 AM, exar...@twistedmatrix.com wrote: On 12:48 pm, c...@msu.edu wrote: [snip] The most *exciting* part of pony-build, apart from the always-riveting spectacle of titus rediscovering problems that buildbot solved 5 years ago, is the loose coupling of recording server to the build slaves and build reporters. ?My plan is to enable a simple and lightweight XML-RPC and/or REST-ish interface for querying the recording server from scripts or other Web sites. ?This has Brett aquiver with anticipation, I gather -- no more visual inspection of buildbot waterfall pages ;) BuildBot has an XML-RPC interface. ?So Brett can probably do what he wants with BuildBot right now. ... but pony-build follows a different model that was just a brief comment to mention that even if both (buildbot + pony-build) support RPC, they are not just «the same» . I'd rather not get into discussions of why my vaporware is going to be way, way better than anything anyone else could possibly do +1 ... I'll be the first one that won't follow it since I have no time for that and my intention was not to suggest that «pb is better than bb» (but if you follow, please do it in a separate thread ;o) @exar...@twistedmatrix.com But BuildBot exists, is deployed, and can be used now, without waiting. +1 as I mentioned before I was not talking about eliminating buildbots (Sorry, I don't really understand what point you were hoping to make with your message, so I just thought I'd follow up in the same style and hope that you'll understand my message even if I don't understand yours :). well, I understand that you don't understand, since I barely understand that I will never be able to understand myself ... :) The only thing I can say so far is that if pb is seriously considered as an option ... then I could give a hand (... and I'll possibly do it anyway , once I have time :-/ ) ... turning myself off ... -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it
Steven D'Aprano schrieb: On Wed, 28 Oct 2009 04:47:59 am Raymond Hettinger wrote: A dict.get() can be meaningfully used in a loop (because the key can vary). A set.get() returns the same value over and over again (because there is no key). I don't believe anyone has requested those semantics. The suggested semantics for set.get() with no arguments, as I understand them, are: (1) it will only fail if the set is empty; (2) it should be efficient; (3) if you call it repeatedly on a set without modifying the set, you will cycle through each element in turn in some unspecified arbitrary order. I don't like this. It gives a set object a hidden state, something that AFAICS no other builtin has. Iterating over an iterable is what iterators are for. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloud build slaves (was Re: Possible language summit topic: buildbots)
On Fri, Oct 30, 2009 at 04:58:29PM -, exar...@twistedmatrix.com wrote: On 04:31 pm, c...@msu.edu wrote: On Fri, Oct 30, 2009 at 04:21:06PM +, Antoine Pitrou wrote: Hello, Sorry for the little redundancy, I would like to underline Jean-Paul's suggestion here: Le Sun, 25 Oct 2009 14:05:12 +, exarkun a ??crit??: I think that money can help in two ways in this case. First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. Amazon's EC2 service is supported out of the box in the latest release. I'm not a PSF member, but it seems to me that the PSF could ask Amazon (or any other virtual machine business anyway) to donate a small number of permanent EC2 instances in order to run buildslaves on. [ ... ] I'm happy to provide VMs or shell access for Windows (XP, Vista, 7); Linux ia64; Linux x86; and Mac OS X. Okay, let's move on this. Martin has, I believe, said that potential slave operators only need to contact him to get credentials for new slaves. Can you make sure to follow up with him to get slaves running on these machines? Or would you rather give out access to someone else and have them do the build slave setup? I think we crossed threads here; I can provide the VMs, and access to them, but I won't (empirically, don't have the regular time available to ;) maintain buildbot buildslaves. You or Antoine or others are welcome to contact me off-list. Just give me an account name and ssh key, and I'll give you login access via tunneled Remote Desktop to the Windows machines. Others have made similar offers. I'll similarly encourage them to take action, then. Do you happen to remember who? Every few months this thread seems to pop up and then fizzles when people realize the level of work and attention involved (- he says cynically) in exploiting the offer of resources; I hope that anyone interested in offering resources will pop their head up again to look around. I hope everyone is on board with the idea of fixing bugs in CPython, either in the actual implementation of features or in the tests for those features. That being the case, the discussion of whether or not the PSF should try to fund such a task is perhaps best discussed on the PSF members list. Sure. --titus -- C. Titus Brown, c...@msu.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
2009/10/30 C. Titus Brown c...@msu.edu: As a counter-offer: Given remote access to however many Windows VMs you want to provide, I'll get them up and running with buildslaves on them. If that requires software such as Visual Studio, I have copies via the MSDN licenses that I am happy to provide. I, too, have MSDN licenses, and I have functioning build environments on all of the VMs (I think -- I've only tested Win XP currently: http://lyorn.idyll.org/ctb/pb-dev/python/detail?result_key=8276 OK, so I guess it's just setting the buildbot stuff up. I also have an OS X 10.5 machine that I can let you into through a firewall; it's building Python 2.7 quite nicely: http://lyorn.idyll.org/ctb/pb-dev/python/detail?result_key=8229 Sorry, I've no experience with OS X at all. Once things are up and running, I'll be prepared to do basic care and feeding of the buildslave, but as my time is limited, it would be nice if others would pitch in to help. I would be somewhat unhappy about giving more than three or four people admin access, but am prepared to lie back and think of England. Greetings from England... :-) I doubt it'll be a huge issue, I just didn't want to end up doing nothing more than delivering 5 more red boxes on the buildbot status page. We can see how it goes. I just think that maintaining the buildbots as more of a community effort means that there's a better chance of issues being fixed quickly. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
On Fri, 30 Oct 2009 at 09:57, Martin v. L?wis wrote: But the real reason for having a buildbot category (or at least a keyword) would be to be able to tag all bugs that are currently making buildbots fail that are _not_ the result of a recent checkin. This would make the task of finding the bugs that need to be cleaned up to stabilize the buildbot fleet easier. I'm currently aware of issues 4970, 3892, and 6462 in this category, and there are a few more that we can/will file if we continue to pay attention to the failure reports now arriving on the irc channel. That's convincing; I've created a buildbot keyword. I gave it the description indicates that tests fail on a buildslave for uncertain reasons If that is indeed the intended purpose of this classification, please keep it in mind when assigning the tag. If I misunderstood the purpose of the keyword, please provide a better description. How about: indicates that related test failures are causing buildbot instability My thought is that sometimes we more-or-less know the reasons for the failure, but for one reason or another we can't fix it immediately, and I'd like to keep such a bug visible when looking at buildbot related issues. IMO it would be no bad thing for this tag to be applied to any issue that is created as a result of an observed test failure on a buildbot. Such an issue should only get created if the person who did the checkin that caused it can't reproduce the problem themselves (ie: they ran the test suite and on their platform it was clean). Now, we know that in practice some bugs show up on buildbots because a committer forgot to run the test suite prior to check in (we all make mistakes), but if such a bug gets tagged 'buildbot' I think that's fine, since it will still be affecting the stability of the buildbots. --David___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Fri, 30 Oct 2009 at 19:46, Paul Moore wrote: 2009/10/30 C. Titus Brown c...@msu.edu: Once things are up and running, I'll be prepared to do basic care and feeding of the buildslave, but as my time is limited, it would be nice if others would pitch in to help. I would be somewhat unhappy about giving more than three or four people admin access, but am prepared to lie back and think of England. Greetings from England... :-) I doubt it'll be a huge issue, I just didn't want to end up doing nothing more than delivering 5 more red boxes on the buildbot status page. We can see how it goes. I just think that maintaining the buildbots as more of a community effort means that there's a better chance of issues being fixed quickly. I'd be happy to help with keeping these (or any other) buildbots running, but I'm not much of a Windows geek (I can work with it, but I know a _lot_ more about Linux). Same goes for OS/X, though since that is unix based I'm a little more comfortable with it. I guess what I'm saying is, if you don't get responses from more Windows-savvy developers, then let me know and I'll be glad to help. That said, the idea of EC2 buildslaves seems pretty attractive... --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it
On Fri, Oct 30, 2009 at 09:37:36PM +0100, Georg Brandl wrote: I don't like this. It gives a set object a hidden state, something that AFAICS no other builtin has. Iterating over an iterable is what iterators are for. It also makes the object thread-unsafe; there's no way for two threads to iterate over it at the same time. It's a terrible idea to introduce new things that won't work under threaded usage. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
exarkun at twistedmatrix.com writes: Is your idea that this would be for tracking issues with the *bots* themselves? That is, not just for tracking cases where some test method fails on a particular bot, but for tracking cases where, say, a bot's host has run out of disk space and cannot run the tests at all? Well the general situation would be slightly easier to appreciate if there was a public medium where buildbot info was exchanged, announcements done, and problems tracked. Some kind of tracker, tracker keyword, mailing-list, or anything else. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot category on the tracker
Well the general situation would be slightly easier to appreciate if there was a public medium where buildbot info was exchanged, announcements done, and problems tracked. Some kind of tracker, tracker keyword, mailing-list, or anything else. As for the tracker keyword - I created one (in case you didn't notice). As for exchanging info: if you talk about the specific Python buildbot installation (instead of info about buildbot, the software), then python-dev is the proper place to exchange info about it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
Since I've never used any such service (cloud-based VMs), I'm not sure what the downsides would be. But it seems to be that it would be at least worth trying. Not sure whether it's still relevant after the offers of individually donated hardware. However, if you want to look into this, feel free to set up EC2 slaves. When it comes to the point of actually having to pay money, please send a request to psf-bo...@python.org (make sure you don't pay anything until the request is approved). Exact processing would have to be decided, then, traditionally, it would be most simple if you could invoice the PSF (IIUC). Giving the PSF as the billing address would probably also work (check with the treasurer). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it
On Sat, 31 Oct 2009 07:27:22 am A.M. Kuchling wrote: On Fri, Oct 30, 2009 at 09:37:36PM +0100, Georg Brandl wrote: I don't like this. It gives a set object a hidden state, something that AFAICS no other builtin has. All objects have a reference count field which is hidden from Python code. The C API for objects has a flags field which specifies whether objects are read-only or read/write from Python code. As of Python 2.6, type objects have an internal method cache. C code can clear it with PyType_ClearCache(), Python codes can't even see it. Lists and dicts pre-allocate extra space, and record hidden state of how much of the space is actually in use. Sets may do the same. File objects may use internal buffers, with all the state that implies. Iterating over an iterable is what iterators are for. set.get(), or set.pick() as Wikipedia calls it, isn't for iterating over sets. It is for getting an arbitrary element from the set. If the requirement that get/pick() cycles through the sets elements is the killer objection to this proposal, I'd be willing to drop it. I thought that was part of the OP's request, but apparently it isn't. I would argue that it's hardly arbitrary if you get the same element every time you call the method, but if others are happy with that behaviour, I'm not going to stand in the way. It also makes the object thread-unsafe; there's no way for two threads to iterate over it at the same time. It's a terrible idea to introduce new things that won't work under threaded usage. I would agree if the purpose of get/pick() was to iterate over the elements of the set, but that's not the purpose. The purpose is to return an arbitrary item each time it is called. If two threads get the same element, that's not a problem; if one thread misses an element because another thread grabbed it first, that's not a problem either. If people prefer a random element instead, I have no problem with that -- personally I think that's overkill, but maybe that's just me. -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it
2009/10/30 Steven D'Aprano st...@pearwood.info: On Sat, 31 Oct 2009 07:27:22 am A.M. Kuchling wrote: On Fri, Oct 30, 2009 at 09:37:36PM +0100, Georg Brandl wrote: I don't like this. It gives a set object a hidden state, something that AFAICS no other builtin has. All objects have a reference count field which is hidden from Python code. The C API for objects has a flags field which specifies whether objects are read-only or read/write from Python code. As of Python 2.6, type objects have an internal method cache. C code can clear it with PyType_ClearCache(), Python codes can't even see it. Lists and dicts pre-allocate extra space, and record hidden state of how much of the space is actually in use. Sets may do the same. File objects may use internal buffers, with all the state that implies. Those are all examples of states which are implementation details. The proposed get() semantics would require that allow implementations keep this state around. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it
On Fri, Oct 30, 2009 at 8:29 PM, Steven D'Aprano st...@pearwood.info wrote: Iterating over an iterable is what iterators are for. set.get(), or set.pick() as Wikipedia calls it, isn't for iterating over sets. It is for getting an arbitrary element from the set. If the requirement that get/pick() cycles through the sets elements is the killer objection to this proposal, I'd be willing to drop it. I thought that was part of the OP's request, but apparently it isn't. I would argue that it's hardly arbitrary if you get the same element every time you call the method, but if others are happy with that behaviour, I'm not going to stand in the way. It's arbitrary in the sense that the API doesn't make any assurances which item the caller will get, not that it's arbitrary for any particular * implementation*. The purpose is to return an arbitrary item each time it is called. If two threads get the same element, that's not a problem; if one thread misses an element because another thread grabbed it first, that's not a problem either. If people prefer a random element instead, I have no problem with that -- personally I think that's overkill, but maybe that's just me. I also think returning a random one is overkill, in the base set. And I'd have the base implementation do the simplest thing possible: return a fixed element (either the first returned when iterated over, or the last stored, or whatever's easiest based on the internals). For me, leaving the choice of *which* element to return on each call is a feature. It allows subclasses to change the behavior to support other use cases, like a random or round-robin behavior. -- Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a setwithoutremoving it
Steven D'Aprano writes: The usual technique people tend to come up with is: x = s.pop() s.add(x) Which strikes many people (including myself) as inelegant. Surely get an element is a more fundamental operation than get an element and remove it? Not in a literal urn or bag. See sampling with replacement in any statistics text. Note that in order to have your cake and eat it too there's an implicit copy operation (although in Python it will be a cheap ref copy rather than an expensive object copy). I'm afraid that the various conflicting intuitions here are all correct. That is going to make design impossible without some compelling use cases. What's wrong with using next()? That is what it's for. You can't call next() on a set. You have to call next(iter(set)). [...] Unless I missed something, this is so unobvious that nobody has suggested it in the thread yet. I've seen it at least twice (Nick suggested it IIRC), although that may have been on Python-Ideas (which is where this thread belongs IMHO). If folks prefer a different name, by all means suggest it. I like the name suggested by Wikipedia, pick. Pick has a connotation of removal in many contexts: Pick a card, any card. Pick it up off the floor (it's not in the set of things on the floor any more). Like get, to me it has a flavor of according to some criterion: a band of picked men. I would expect a pick or get operation to take an optional predicate. But then TOOWTDI is for x in container: if predicate(x): break else: x = default_x# or perhaps raise in cases where there # theoretically should be an element My reasoning for the given behaviour is as follows: * If you want the same element twice from a set, Nobody is asking for that AFAICS. The use case I have so far observed people to have in mind is a cast from an equivalence class to a representative member. They don't care whether the member is the same or not. If they wanted iterator behavior, getting it would not be a problem. next(iter(foo_set)) is TOOWTDI. If they wanted cyclical behavior, itertools.cycle. If they wanted random behavior, make a list and choose from it. In one leading subcase, the equivalence class is a singleton. In this use case what people really want, I suspect, is behavior like Python strings and characters: a singleton set should provide the same operations as its unique element and vice versa. So having get() return the same value each time is not useful or necessary. Which is not the point. The point is that having get return a different value if possible is not useful or necessary in at least some of the leading use cases. OTOH, we don't know enough about weird use cases where having a different value returned is important to decide how that should be done, where weird means outside of the list of already Obvious Ways. No. Since sets are unordered, there's no way to seek to a specific element. I think people will realize that in fact *these* sets are ordered and they will demand a seek function for various practical purposes. Sorry for so many questions, but I honestly think there are too many unresolved design issues. I have to agree with Raymond. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
