Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
John Arbash Meinel wrote: res = heads(node1, node2) if len(res) == 1: # What is the 'obvious' way to get the node out? I posit that there *isn't* an obvious way to get the single item out of a 1-entry frozenset. for x in res: break list(res)[0] set(res).pop() iter(res).next() [x for x in res][0] x, = res # I didn't think of this one before recently Are all answers, but none of them I would consider *obvious*. And from my SQL-hacking experience: x = min(s) --Scott David Daniels scott.dani...@acm.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
> We were using sets to track the tips of a graph, and to compare whether > one node was an ancestor of another one. We were caching that answer > into frozensets, since that made them immutable. If > > res = heads(node1, node2) > if len(res) == 1: > # What is the 'obvious' way to get the node out? What specifically is that that you want to do in this case? IIUC, the possible result could be that either node1 is an ancestor of node2, or vice versa. So I would write that as if len(res) == 1: if node1 in res: # action to take if node1 is the head else: # action to take if node2 is the head else: # they are unrelated In any case, this is a use case different from "work queue" (AFAICT), so I'm unsure why you responded to my message where Florian and me were talking about work queues. > res.get() would be a fairly obvious way to do it. Enough that I would > probably never have gone searching for any of the other answers. Though > personally, I think I would call it "set.peek()", but the specific name > doesn't really matter to me. Somebody proposed to call it .any(); this I like best (except that one might expect that any takes a predicate as the argument). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
Antoine Pitrou wrote: Hello there, The last couple of days I've been working on an experimental rewrite of the GIL. Since the work has been turning out rather successful (or, at least, not totally useless and crashing!) I thought I'd announce it here. I am curious as to whether the entire mechanism is or can be turned off when not needed -- when there are not threads (other than the main, starting thread)? tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
> Martin v. Löwis wrote: >>> Hmm, perhaps when using sets as work queues? >> >> A number of comments: >> >> - it's somewhat confusing to use a set as a *queue*, given >> that it won't provide FIFO semantics. >> - there are more appropriate and direct container structures >> available, including a dedicated Queue module (even though >> this might be doing to much with its thread-safety). >> - if you absolutely want to use a set as a work queue, >> then the .pop() method should be sufficient, right? >> >> Regards, >> Martin > > We were using sets to track the tips of a graph, and to compare whether > one node was an ancestor of another one. We were caching that answer > into frozensets, since that made them immutable. If > > res = heads(node1, node2) > if len(res) == 1: > # What is the 'obvious' way to get the node out? > > I posit that there *isn't* an obvious way to get the single item out of > a 1-entry frozenset. > > for x in res: break > list(res)[0] > set(res).pop() > iter(res).next() > [x for x in res][0] > x, = res # I didn't think of this one before recently > > Are all answers, but none of them I would consider *obvious*. At the > least, none of them are obviously better than another, so you look into > the performance characteristics to give you a reason to pick one over > the other. > > res.get() would be a fairly obvious way to do it. Enough that I would > probably never have gone searching for any of the other answers. Though > personally, I think I would call it "set.peek()", but the specific name > doesn't really matter to me. > > John > When I first wrote Graphine (graph library), I did something very similar to the last solution. The code has since been rewritten to avoid the issue, but it would be nice if it didn't have to be the next time it comes up- the comma is easy to miss, and while the results are common-sense once you get what the line is doing, it doesn't lend itself to immediate comprehension. Geremy Condra ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Martin v. Löwis wrote: >> Hmm, perhaps when using sets as work queues? > > A number of comments: > > - it's somewhat confusing to use a set as a *queue*, given > that it won't provide FIFO semantics. > - there are more appropriate and direct container structures > available, including a dedicated Queue module (even though > this might be doing to much with its thread-safety). > - if you absolutely want to use a set as a work queue, > then the .pop() method should be sufficient, right? > > Regards, > Martin We were using sets to track the tips of a graph, and to compare whether one node was an ancestor of another one. We were caching that answer into frozensets, since that made them immutable. If res = heads(node1, node2) if len(res) == 1: # What is the 'obvious' way to get the node out? I posit that there *isn't* an obvious way to get the single item out of a 1-entry frozenset. for x in res: break list(res)[0] set(res).pop() iter(res).next() [x for x in res][0] x, = res # I didn't think of this one before recently Are all answers, but none of them I would consider *obvious*. At the least, none of them are obviously better than another, so you look into the performance characteristics to give you a reason to pick one over the other. res.get() would be a fairly obvious way to do it. Enough that I would probably never have gone searching for any of the other answers. Though personally, I think I would call it "set.peek()", but the specific name doesn't really matter to me. John =:-> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrlB4MACgkQJdeBCYSNAAN0lQCgrtyXWlqIbjj01qB4AKPhKrMq QH8An0z2gCWZHoceEJsqRJOUdEl/VLTB =fJXI -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On 01:28 am, db3l@gmail.com wrote: exar...@twistedmatrix.com writes: This sounds like something that should be reported upstream. Particularly if you know how to reproduce it. Has it been? No, largely because I can't reproduce it at all. It's happened maybe 4-5 times in the past 2 years or so. All that I see is that my end looks good yet the master end seems not to be dispatching jobs (it never shows an explicit disconnect for my slave though). My best guess is that something disrupted the TCP connection, and that the slave isn't doing anything that would let it know its connection was dropped. Although I thought there were periodic pings even from the slave side. Given the frequency, it's not quite high priority to me, though having the master let the owner of a slave know when it's down would help cut down on lost availability due to this case, so I suppose I could suggest that feature to the buildbot developers. This feature exists, at least. BuildBot can email people when slaves are offline for more than some configured time limit. I'm not sure if the CPython master is configured to do this or not. It's easy to set up if not, the BuildSlave initializer accepts a list of email addresses that will be notified when that slave goes offline, notify_on_missing: http://buildbot.net/apidocs/buildbot.buildslave.AbstractBuildSlave- class.html#__init__ Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
MRAB writes: > Couldn't you write a script to check the status periodically? Sure, I suppose scraping the web status page would work. If it happened frequently I'd probably be forced to do something like that, but it's relatively low frequency (though I guess it does have a big impact in terms of availability) makes it hard to dedicate time to that compared to my "real world" work :-) -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
exar...@twistedmatrix.com writes: > This sounds like something that should be reported > upstream. Particularly if you know how to reproduce it. Has it been? No, largely because I can't reproduce it at all. It's happened maybe 4-5 times in the past 2 years or so. All that I see is that my end looks good yet the master end seems not to be dispatching jobs (it never shows an explicit disconnect for my slave though). My best guess is that something disrupted the TCP connection, and that the slave isn't doing anything that would let it know its connection was dropped. Although I thought there were periodic pings even from the slave side. Given the frequency, it's not quite high priority to me, though having the master let the owner of a slave know when it's down would help cut down on lost availability due to this case, so I suppose I could suggest that feature to the buildbot developers. -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On 25 Oct, 09:36 pm, db3l@gmail.com wrote: I think the other issue most likely to cause a perceived "downtime" with the Windows build slave that I've had a handful of cases over the past two years where the build slave appears to be operating properly, but the master seems to just queue up jobs as if it were down. The slave still shows an established TCP link to the master, so I generally only catch this when I happen to peek at the status web page, or catch a remark here on python-dev, so that can reduce availability. This sounds like something that should be reported upstream. Particularly if you know how to reproduce it. Has it been? Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
David Bolen wrote: [snip] I think the other issue most likely to cause a perceived "downtime" with the Windows build slave that I've had a handful of cases over the past two years where the build slave appears to be operating properly, but the master seems to just queue up jobs as if it were down. The slave still shows an established TCP link to the master, so I generally only catch this when I happen to peek at the status web page, or catch a remark here on python-dev, so that can reduce availability. [snip] Couldn't you write a script to check the status periodically? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
On Mon, 26 Oct 2009 05:43:52 am Martin v. Löwis wrote:
> What Alexander wants is that the set type also becomes useful for
> storing canonical representations. I find that inappropriate, since
> dicts already provide the one obvious way to do it.
Only because dicts came first in Python rather than sets. There's
nothing inherently obvious about storing the canonical representation
of an object *twice* (although I'm not Dutch). A more natural
representation would be to store the canonical representation once.
An abstract intern operation might be written:
if s equals an element in the data store
return the element which s equals
otherwise
insert s into the data store and return s
Notice we don't say:
if s equals an element in the data store
return the value associated with the element which s equals
which is what the dict-based solution does.
We can implement that abstract algorithm using lists. The algorithm is
exactly the same as it is for dicts, except that search/retrieval
becomes two operations rather than one:
_cache = ['x', 'spam', 'parrot']
def intern(s):
try:
s = _cache[ _cache.index(s) ]
except ValueError:
_cache.append(s)
return s
We don't do this because O(N) searching is too expensive. Using a dict
{s: s} is a workaround for the lack of a data structure that combines
O(1) searching and ability to retrieve the element just found.
Memory is cheap, but not so cheap that doubling the size of a data
structure (two pointers per element for {s:s} versus one for {s}) is
insignificant. Part of the reason we intern in the first place is to
save memory. We shouldn't dismiss this out of hand based on the
argument that retrieval from a set is insufficiently pure. As soon as
you allow iteration over sets, you have allowed retrieval.
--
Steven D'Aprano
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
> > [SNIP - a lot of detail on what sounds like a good design] > > Now what remains to be done? > > Having other people test it would be fine. Even better if you have an > actual multi-threaded py3k application. But ccbench results for other > OSes would be nice too :-) > (I get good results under the Windows XP VM but I feel that a VM is not > an ideal setup for a concurrency benchmark) > > Of course, studying and reviewing the code is welcome. As for > integrating it into the mainline py3k branch, I guess we have to answer > these questions: > - is the approach interesting? (we could decide that it's just not worth > it, and that a good GIL can only be a dead (removed) GIL) > I think it's worth it. Removal of the GIL is a totally open-ended problem with no solution in sight. This, on the other hand, is a performance benefit now. I say move forward with this. If it happens to be short-lived because some actually figures out how to remove the GIL then great, but is that really going to happen between now and Python 3.2? I doubt it. > - is the patch good, mature and debugged enough? > - how do we deal with the unsupported platforms (POSIX and Windows > support should cover most bases, but the fate of OS/2 support depends on > Andrew)? > > It's up to Andrew to get the support in. While I have faith he will, this is why we have been scaling back the support for alternative OSs for a while and will continue to do so. I suspect the day Andrew stops keeping up will be the day we push to have OS/2 be externally maintained. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
ssteinerX gmail.com gmail.com> writes: > On Oct 25, 2009, at 5:43 PM, Martin v. Löwis wrote: > > > Only turning on the slave occasionally makes it useless. > > For certain use cases; not mine. Let's say that for the use case we are talking here (this is python-dev), Martin's statement holds true. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Oct 25, 2009, at 5:43 PM, Martin v. Löwis wrote: Only turning on the slave occasionally makes it useless. For certain use cases; not mine. S ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
> It's not so much that I *require* the slave to shut down, more that > I'm not sure how well I'll be able to ensure that it's up all the > time, and I'm trying to understand the implications of that. My basic > impression is that it's not really going to work, unfortunately. There is a significant difference between "not able to ensure that it is up all the time", and "it is down most of the time, and only up once a day for a short period of time". Regular and irregular short downtimes are no problem at all. When the slave comes back, it will pick up pending work. Longer scheduled downtimes (e.g. for vacations) are acceptable. Only turning on the slave occasionally makes it useless. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
"Martin v. Löwis" writes: > The remaining issue is the popups; if a process still has a popup, > you can't even terminate it properly. There are two kinds of popups: > system-generated ones, and CRT-generated ones. For the CRT ones, we > once had a way to turn them off, but I'm not sure whether that mechanism > might have been removed. For the system messages, there is a way to > turn them off in the parent process. David Bolen (IIRC) had developed > a patch, but I think this patch only runs on his system(s). Yes, process-stopping dialogs have probably been the single most annoying issue over time running a Windows build slave - certainly from my perspective in terms of maintenance and detection they have taken up the largest amount of time. I believe the CRT disabling is still active in the 3.x branches (the "-n" flag to regrtest in the buildbot's test.bat), after you restored it this past March (it had regressed during an earlier py3k branch set of patches and caused a bunch of problems for a bit) but not in the 2.x branches or trunk. So there's still a bit of exposure there and I'd certainly be in favor of porting the regrtest -n support over to all current development branches. I think the other issue most likely to cause a perceived "downtime" with the Windows build slave that I've had a handful of cases over the past two years where the build slave appears to be operating properly, but the master seems to just queue up jobs as if it were down. The slave still shows an established TCP link to the master, so I generally only catch this when I happen to peek at the status web page, or catch a remark here on python-dev, so that can reduce availability. My build slave (based on 0.7.5 I think) runs with local patches to: 1. Protect again Win32 pop-up error boxes in child processes. 2. A fix for a recursive chaining of Twisted Deferreds during uploads which could break for large file transfers. This only came up when my build slave was generating daily MSI builds and uploading them to the master. 3. Handle clock jumps. It's a general flaw in the presence of system clock adjustments, but I only encountered it while in my FreeBSD build slave under VMWare with a Linux host. (2) and (3) are useful, but not likely to be an issue with most build slaves in normal operation. (2) probably isn't needed on my end any more now that the daily MSI builds aren't run, and it's possible that it got corrected in later buildbot updates, since I did report it on the development list at the time. (1) is a pretty trivial patch, but added a dependency on pywin32, so passing it back up to the buildbot maintainers (back in 2007) stalled while determining if that was ok, and I don't I ever closed the loop. I did later make it fall back to ctypes if pywin32 was missing, but then I think buildbot was using a minimum of Python 2.3 at the time, so even ctypes was a new dependency. Anyway, it became less crucial when Python's regrtest started executing similar code, though the buildbot patch covers anything run under it and not just the python process. I'd of course be happy to pass along the patch to anyone interested. I believe that Thomas Heller had run his Windows buildbot with some similar local code, but implemented with a modified buildbot script for building Python, rather than tweaking buildbot itself. Of course, the patch only protects against system pop-ups - it can't control the CRT assertion dialogs when Python is built in debug mode, which is why I've argued in the past that the test process for Python should ensure those are disabled. The CRT errors themselves are still important, but can be redirected to stderr rather than a blocking GUI dialog. -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Oct 25, 2009, at 3:06 PM, Martin v. Löwis wrote: (*) it may help if Buildbot would create a Win32 job object, and then use TerminateJobObject. Contributions are welcome. Some work has already been done on this, but it needs help. At the root it's a Twisted issue: http://twistedmatrix.com/trac/ticket/2726 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
2009/10/25 "Martin v. Löwis" : >> I've been trying to get some feedback about firing up buildbots on Cloud >> Servers for a while now and haven't had much luck. I'd love to find a >> way of having buildbots come to life, report to the mother ship, do the >> build, then go away 'till next time they're required. > > I'm not quite sure whom you have been trying to get feedback from, and > can't quite picture your proposed setup from above description. Sorry, feedback was the wrong word. I've been digging round the documentation I've been able to find and looking into what's needed to set up a slave. > In any case, ISTM that your approach isn't compatible with how buildbot > works today (not sure whether you are aware of that): a build slave > needs to stay connected all the time, so that the build master can > trigger a build when necessary. > > So if your setup requires the slaves to shut down after a build, I don't > think this can possibly work. It's not so much that I *require* the slave to shut down, more that I'm not sure how well I'll be able to ensure that it's up all the time, and I'm trying to understand the implications of that. My basic impression is that it's not really going to work, unfortunately. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Buildbot alternate renderings
Triggered by the recent discussion, I looked at the changes in status display that the buildbot people have implemented recently. I added a few links into alternate renderings; see http://www.python.org/dev/buildbot/ Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [TIP] Possible language summit topic: buildbots
> Brett actually wants web hooks so pony-build will ping an App Engine web > app when there is more data, ala PubSubHubbub. Or hell, just have > pony-build have an Atom feed with updates and simply use PuSH. In other > words I want to be told when there is an update, not have to poll to > find out. Not sure what exactly it is that Brett wants to do, but perhaps Brett could take a look at http://www.python.org/dev/buildbot/all/atom As JP says, there is also XML-RPC (at all/xmlrpc) For a true push notifications: buildbot sends messages into an IRC channel - not sure whether an App Engine App could listen to that. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
> The most *exciting* part of pony-build, apart from the always-riveting > spectacle of "titus rediscovering problems that buildbot solved 5 years ago", > is the loose coupling of recording server to the build slaves and build > reporters. My plan is to enable a simple and lightweight XML-RPC and/or > REST-ish interface for querying the recording server from scripts or other Web > sites. This has Brett aquiver with anticipation, I gather -- no more visual > inspection of buildbot waterfall pages ;) If that's something that people want to have, then buildbot could also provide it, of course. Do you have a spec of the interface somewhere? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Oct 25, 2009, at 3:35 PM, Martin v. Löwis wrote: I don't need to know that it works on every checkin For us, that is a fairly important requirement, though. Reports get more and more useless if they aren't instantaneous. Sometimes, people check something in just to see how the build slaves react. Understood -- that's why I mentioned it. This is a different use-case. It may still have some use for Python itself, but my idea is more for testing things like libraries where the developer may only work on or have one platform and may want to test installing on other platforms and Python versions during development and/or before release. S ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
> I don't need to know that it works on every checkin For us, that is a fairly important requirement, though. Reports get more and more useless if they aren't instantaneous. Sometimes, people check something in just to see how the build slaves react. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
> This is supported in recent versions of BuildBot with a special kind of > slave: > > http://djmitche.github.com/buildbot/docs/0.7.11/#On_002dDemand- > _0028_0022Latent_0022_0029-Buildslaves Interesting. Coming back to "PSF may spend money", let me say this: If somebody would volunteer to set up slaves in EC2, and operate them, I'm fairly certain that the PSF would pay the bill. It might be useful to have two people operating them, so that the knowledge and the load is shared. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
> it shouldn't be difficult to cobble together a build script that spins up a > buildslave on EC2 and runs the tests there; I wrote something similar a > few years ago for an infrequently connected home machine. Ok - so it would be the master running this script? Sounds reasonable to me. As for EC2 (and other cloud providers); I'm somewhat skeptical about platform coverage, also. How many different processors and operating systems can they possibly support? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
> I think that money can help in two ways in this case. > > First, there are now a multitude of cloud hosting providers which will > operate a slave machine for you. BuildBot has even begun to support > this deployment use-case by allowing you to start up and shut down vms > on demand to save on costs. Amazon's EC2 service is supported out of > the box in the latest release. Here I'm skeptical. I think we can find people donating always-online machines still; no need to throw donated money to Amazon. > Second, there are a number of active BuildBot developers. One of them > has even recently taken a contract from Mozilla to implement some non- > trivial BuildBot enhancements. I think it very likely that he would > consider taking such a contract from the PSF for whatever enhancements > would help out the CPython buildbot. That could indeed be interesting, assuming we had a clear requirement. But then, most of us can "easily" fix things in buildbot themselves - this is python-dev, after all. >> This is something that only the slave admins can answer. I don't think >> it's difficult - it's just that people are really unlikely to contribute >> to the same thing over a period of five years at a steady rate. So we >> need to make sure to find replacements when people drop out. > > This is a good argument for VMs. Not really - you still would need somebody to manage them. > It's certainly *possible* to chase an > ever changing set of platforms, but it strikes me as something of a > waste of time. Hmm - can you really get "strange" operating systems "in the cloud"? Some of the operating systems that we would like to test don't even support VMs. > To me, that raises the question of why people aren't more concerned with > the status of the build system. Shouldn't developers care if the code > they're writing works or not? I think there are two issues here: 1. some developers actually *don't* care to much whether their code works in all cases. If it fails on some strange platform they never heard of (such as "Solaris", or "Windows"), they are not bothered by the failure. Or, if they care, they still don't know what to do about the failure. 2. the buildbots sometimes report false positives. Some tests fail in a non-repeatable fashion, only on selected systems. So when you change something, the tests break, and you cannot really see how this could be possibly related to your change. Then you start ignoring these reports - both the bogus ones, and the real ones. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
Martin v. Löwis v.loewis.de> writes: > > If it's offline too often, I'm skeptical that it would be useful. If > you report breakage after a day, then it will be difficult to attribute > this to a specific commit. It is most useful to have continuous > integration if error reports are instantaneous. Not only, but it's useful to have a stable set of buildbots, so that when a test fails, you know whether it's a new problem caused by a recent revision, or rather an already existing problem on this platform. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Oct 25, 2009, at 2:32 PM, Martin v. Löwis wrote: I've been trying to get some feedback about firing up buildbots on Cloud Servers for a while now and haven't had much luck. I'd love to find a way of having buildbots come to life, report to the mother ship, do the build, then go away 'till next time they're required. I'm not quite sure whom you have been trying to get feedback from, and can't quite picture your proposed setup from above description. I posted a couple of messages on testing-in-python, and have sent around some queries to others that I know are using buildbot type setups with various tools/platforms, not necessarily Python. In any case, ISTM that your approach isn't compatible with how buildbot works today (not sure whether you are aware of that): a build slave needs to stay connected all the time, so that the build master can trigger a build when necessary. So if your setup requires the slaves to shut down after a build, I don't think this can possibly work. It can't possibly work within the way the Python buildbot structure currently works, as I understand it so far. What I'm implementing is less of a 'continuous integration' tool like you would use for something like Python itself, and more of a testing tool for things that have to be installed on multiple versions of Python, on multiple platforms. I don't need to know that it works on every checkin, just every once in a while I'd like to start from scratch and make sure everything still works on all supported versions of Python on all the platforms I test on and cloud servers are great for this since I'll usually only need them for an hour or so. S ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Reworking the GIL
Hello there,
The last couple of days I've been working on an experimental rewrite of
the GIL. Since the work has been turning out rather successful (or, at
least, not totally useless and crashing!) I thought I'd announce it
here.
First I want to stress this is not about removing the GIL. There still
is a Global Interpreter Lock which serializes access to most parts of
the interpreter. These protected parts haven't changed either, so Python
doesn't become really better at extracting computational parallelism out
of several cores.
Goals
-
The new GIL (which is also the name of the sandbox area I've committed
it in, "newgil") addresses the following issues :
1) Switching by opcode counting. Counting opcodes is a very crude way of
estimating times, since the time spent executing a single opcode can
very wildly. Litterally, an opcode can be as short as a handful of
nanoseconds (think something like "... is not None") or as long as a
fraction of second, or even longer (think calling a heavy non-GIL
releasing C function, such as re.search()). Therefore, releasing the GIL
every 100 opcodes, regardless of their length, is a very poor policy.
The new GIL does away with this by ditching _Py_Ticker entirely and
instead using a fixed interval (by default 5 milliseconds, but settable)
after which we ask the main thread to release the GIL and let another
thread be scheduled.
2) GIL overhead and efficiency in contended situations. Apparently, some
OSes (OS X mainly) have problems with lock performance when the lock is
already taken: the system calls are heavy. This is the "Dave Beazley
effect", where he took a very trivial loop, therefore made of very short
opcodes and therefore releasing the GIL very often (probably 10
times a second), and runs it in one or two threads on an OS with poor
lock performance (OS X). He sees a 50% increase in runtime when using
two threads rather than one, in what is admittedly a pathological case.
Even on better platforms such as Linux, eliminating the overhead of many
GIL acquires and releases (since the new GIL is released on a fixed time
basis rather than on an opcode counting basis) yields slightly better
performance (read: a smaller performance degradation :-)) when there are
several pure Python computation threads running.
3) Thread switching latency. The traditional scheme merely releases the
GIL for a couple of CPU cycles, and reacquires it immediately.
Unfortunately, this doesn't mean the OS will automatically switch to
another, GIL-awaiting thread. In many situations, the same thread will
continue running. This, with the opcode counting scheme, is the reason
why some people have been complaining about latency problems when an I/O
thread competes with a computational thread (the I/O thread wouldn't be
scheduled right away when e.g. a packet arrives; or rather, it would be
scheduled by the OS, but unscheduled immediately when trying to acquire
the GIL, and it would be scheduled again only much later).
The new GIL improves on this by combinating two mechanisms:
- forced thread switching, which means that when the switching interval
is terminated (mentioned in 1) and the GIL is released, we will force
any of the threads waiting on the GIL to be scheduled instead of the
formerly GIL-holding thread. Which thread exactly is an OS decision,
however: the goal here is not to have our own scheduler (this could be
discussed but I wanted the design to remain simple :-) After all,
man-years of work have been invested in scheduling algorithms by kernel
programming teams).
- priority requests, which is an option for a thread requesting the GIL
to be scheduled as soon as possible, and forcibly (rather than any other
threads). This is meant to be used by GIL-releasing methods such as
read() on files and sockets. The scheme, again, is very simple: when a
priority request is done by a thread, the GIL is released as soon as
possible by the thread holding it (including in the eval loop), and then
the thread making the priority request is forcibly scheduled (by making
all other GIL-awaiting threads wait in the meantime).
Implementation
--
The new GIL is implemented using a couple of mutexes and condition
variables. A {mutex, condition} pair is used to protect the GIL itself,
which is a mere variable named `gil_locked` (there are a couple of other
variables for bookkeeping). Another {mutex, condition} pair is used for
forced thread switching (described above). Finally, a separate mutex is
used for priority requests (described above).
The code is in the sandbox:
http://svn.python.org/view/sandbox/trunk/newgil/
The file of interest is Python/ceval_gil.h. Changes in other files are
very minimal, except for priority requests which have been added at
strategic places (some methods of I/O modules). Also, the code remains
rather short, while of course being less trivial than the old one.
NB : this is a branch of py3k. There should be no real difficulty
porting it back to trunk
Re: [Python-Dev] Possible language summit topic: buildbots
> OK, sounds useful. If I'm offline for a while, do multiple builds get > queued, or only the "last" one? IIRC, it will only build the last one, then with a huge blame list. > If the former, I can imagine coming > back to a pretty huge load if the slave breaks while I'm on holiday > :-( If it's offline too often, I'm skeptical that it would be useful. If you report breakage after a day, then it will be difficult to attribute this to a specific commit. It is most useful to have continuous integration if error reports are instantaneous. > I should look all of this up somewhere. Is there a reference to > buildbot for slave maintainers? Are there any specifics for Python > slaves that I should refer to? Hmm. I thought I had send you this before: http://wiki.python.org/moin/BuildbotOnWindows > (From what I've been able to find, it seems to me that setting up a > slave first requires getting things sorted with the admins, which > sadly precludes experimenting to find things out - I can understand > why the python admins don't want people "playing" on the live buildbot > infrastructure, though :-)) That's really not an issue. Feel free to play as much as you want, with the live infrastructure. You can't break anything doing so (perhaps except for spamming the mailing list with faulty reports). If you then decide to withdraw your offer, that's fine as well (just make sure to notify us instead of just silently taking the slave down). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [TIP] Possible language summit topic: buildbots
On Sun, Oct 25, 2009 at 07:13, wrote: > On 12:48 pm, c...@msu.edu wrote: > >> >> [snip] >> >> >> The most *exciting* part of pony-build, apart from the always-riveting >> spectacle of "titus rediscovering problems that buildbot solved 5 years >> ago", >> is the loose coupling of recording server to the build slaves and build >> reporters. My plan is to enable a simple and lightweight XML-RPC and/or >> REST-ish interface for querying the recording server from scripts or other >> Web >> sites. This has Brett aquiver with anticipation, I gather -- no more >> visual >> inspection of buildbot waterfall pages ;) >> > > BuildBot has an XML-RPC interface. So Brett can probably do what he wants > with BuildBot right now. > Brett actually wants web hooks so pony-build will ping an App Engine web app when there is more data, ala PubSubHubbub. Or hell, just have pony-build have an Atom feed with updates and simply use PuSH. In other words I want to be told when there is an update, not have to poll to find out. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
> As I have no specific experience maintaining any of the CPython build > slaves, I can't speak to any maintenance issues which these slaves have > encountered. I would expect that they are as minimal as the issues I > have encountered maintaining slaves for other projects, but perhaps this > is wrong. I do recall that there were some win32 issues (discussed on > this list, I think) quite a while back, but I think those were resolved. > I haven't heard of any other issues since then. Only partially. One old issue was that previous builds would not complete, keeping the executable files open, preventing further runs. Buildbot is supposed to kill a build, but only kills the parent process (as it is really difficult to kill the entire process tree (*)). We work around this by explicitly killing any stale Python processes at the beginning of a new build. The remaining issue is the popups; if a process still has a popup, you can't even terminate it properly. There are two kinds of popups: system-generated ones, and CRT-generated ones. For the CRT ones, we once had a way to turn them off, but I'm not sure whether that mechanism might have been removed. For the system messages, there is a way to turn them off in the parent process. David Bolen (IIRC) had developed a patch, but I think this patch only runs on his system(s). Regards, Martin (*) it may help if Buildbot would create a Win32 job object, and then use TerminateJobObject. Contributions are welcome. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
Antoine Pitrou wrote: >> For a), I think we can solve this only by redundancy, i.e. create more >> build slaves, hoping that a sufficient number would be up at any point >> in time. > > We are already doing this, aren't we? > http://www.python.org/dev/buildbot/3.x/ > > It doesn't seem to work very well, it's a bit like a Danaides vessel. Both true. However, it seems that Mark is unhappy with the current set of systems, so we probably need to do it again. > Well, to be fair, buildbots breaking also happens much more frequently > (perhaps one or two orders of magnitude) than the SVN server or the Web > site going down. Maintaining them looks like a Sisyphean task, and nobody > wants that. It only looks so. It is like any server management task - it takes constant effort. However, it is not Sisyphean (feeling Greek today, ain't you :-); since you actually achieve something. It's not hard to restart a buildbot when it has crashed, and it gives a warm feeling of having achieved something. > I don't know what kind of machines are the current slaves, but if they > are 24/7 servers, isn't it a bit surprising that the slaves would go down > so often? Is the buildbot software fragile? Not really. It sometimes happens that the slaves don't reconnect after a master restart, but more often, it is just a change on the slave side that breaks it (such as a reboot done to the machine, and not having the machine configured to restart the slave after the reboot). > Does it require a lot of > (maintenance, repair) work from the slave owners? On Unix, not really. On Windows, there is still the issue that sometimes, some error message pops up which you need to click away. Over several builds, you may find that you have to click away dozens of such messages. This could use some improvement. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
> Hmm, perhaps when using sets as work queues? A number of comments: - it's somewhat confusing to use a set as a *queue*, given that it won't provide FIFO semantics. - there are more appropriate and direct container structures available, including a dedicated Queue module (even though this might be doing to much with its thread-safety). - if you absolutely want to use a set as a work queue, then the .pop() method should be sufficient, right? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On 06:32 pm, mar...@v.loewis.de wrote: I've been trying to get some feedback about firing up buildbots on Cloud Servers for a while now and haven't had much luck. I'd love to find a way of having buildbots come to life, report to the mother ship, do the build, then go away 'till next time they're required. I'm not quite sure whom you have been trying to get feedback from, and can't quite picture your proposed setup from above description. In any case, ISTM that your approach isn't compatible with how buildbot works today (not sure whether you are aware of that): a build slave needs to stay connected all the time, so that the build master can trigger a build when necessary. So if your setup requires the slaves to shut down after a build, I don't think this can possibly work. This is supported in recent versions of BuildBot with a special kind of slave: http://djmitche.github.com/buildbot/docs/0.7.11/#On_002dDemand- _0028_0022Latent_0022_0029-Buildslaves Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
> If you actually care about that aspect of the semantics for a particular > application, it would be far clearer to use a full dict() and live with > the additional memory usage. While I can see how saving that > pointer-per-entry might make sense in some circumstances, for most I > would see it as a needlessly confusing micro-optimisation. That's exactly my view. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
> Given Alexander's premise, I agree with your response. But his premise > is wrong. Python's current builtin set class maps abstract equivalence > classes to representative members. And this *is* useful. Mapping > arbitrary members of such classes to representative members, sometimes > called 'canonical', is a standard problem/goal in math. String interning > is an application of this idea. Right. However, this is conceptually a function. In some cases (like string interning), it is mutable (over time) and finite (at any point in time). We already have a data type that can perfectly represent mutable finite funcitons, namely dictionaries. And indeed, interning is implemented by a dictionary. What Alexander wants is that the set type also becomes useful for storing canonical representations. I find that inappropriate, since dicts already provide the one obvious way to do it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Sun, Oct 25, 2009 at 07:32:52PM +0100, "Martin v. L?wis" wrote: > > I've been trying to get some feedback about firing up buildbots on Cloud > > Servers for a while now and haven't had much luck. I'd love to find a > > way of having buildbots come to life, report to the mother ship, do the > > build, then go away 'till next time they're required. > > I'm not quite sure whom you have been trying to get feedback from, and > can't quite picture your proposed setup from above description. > > In any case, ISTM that your approach isn't compatible with how buildbot > works today (not sure whether you are aware of that): a build slave > needs to stay connected all the time, so that the build master can > trigger a build when necessary. Hi Martin, it shouldn't be difficult to cobble together a build script that spins up a buildslave on EC2 and runs the tests there; I wrote something similar a few years ago for an infrequently connected home machine. --titus -- C. Titus Brown, c...@msu.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
> I've been trying to get some feedback about firing up buildbots on Cloud > Servers for a while now and haven't had much luck. I'd love to find a > way of having buildbots come to life, report to the mother ship, do the > build, then go away 'till next time they're required. I'm not quite sure whom you have been trying to get feedback from, and can't quite picture your proposed setup from above description. In any case, ISTM that your approach isn't compatible with how buildbot works today (not sure whether you are aware of that): a build slave needs to stay connected all the time, so that the build master can trigger a build when necessary. So if your setup requires the slaves to shut down after a build, I don't think this can possibly work. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
* James Y. Knight: > On Oct 25, 2009, at 2:50 AM, Terry Reedy wrote: > >> Alex Martelli wrote: >>> Next(s) would seem good... >> >> That does not work. It has to be next(iter(s)), and that has been >> tried and eliminated because it is significantly slower. > > But who cares about the speed of getting an arbitrary element from a > set? How can it *possibly* be a problem in a real program? Hmm, perhaps when using sets as work queues? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
2009/10/25 : > If you run a build slave and it's offline when a build is requested, the > build will be queued and run when the slave comes back online. So if the > CPython developers want to work this way (I wouldn't), then we don't need > pony-build; BuildBot will do just fine. OK, sounds useful. If I'm offline for a while, do multiple builds get queued, or only the "last" one? If the former, I can imagine coming back to a pretty huge load if the slave breaks while I'm on holiday :-( I should look all of this up somewhere. Is there a reference to buildbot for slave maintainers? Are there any specifics for Python slaves that I should refer to? (From what I've been able to find, it seems to me that setting up a slave first requires getting things sorted with the admins, which sadly precludes experimenting to find things out - I can understand why the python admins don't want people "playing" on the live buildbot infrastructure, though :-)) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On 05:47 pm, p.f.mo...@gmail.com wrote: 2009/10/25 : Perhaps this is a significant portion of the problem. Maintaining a build slave is remarkably simple and easy. I maintain about half a dozen slaves and spend at most a few minutes a month operating them. Actually setting one up in the first place might take a bit longer, since it involves installing the necessary software and making sure everything's set up right, but the actual slave configuration itself is one command: buildbot create-slavepassword> Perhaps this will help dispel the idea that it is a serious undertaking to operate a slave. The real requirement which some people may find challenging is that the slave needs to operate on a host which is actually online almost all of the time. If you don't such a machine, then there's little point offering to host a slave. I have been seriously considering setting up one or more buildslaves for a while now. However, my biggest issue is that they would be running as VMs on my normal PC, which means that it's the issue of keeping them continually online that hurts me. If I could (say) just fire the slaves up for a set period, or fire them up, have them do a build and report back, and then shut down, that would make my life easier (regular activities rather than ongoing sysadmin works better for me). It sounds like a buildslave isn't really what I should be looking at. Maybe Titus' push model pony-build project would make more sense for me. Maybe. I wonder if Titus' "push model" (I don't really understand this term in this context) makes sense for continuous integration at all, though. As a developer, I don't want to have access to build results across multiple platforms when someone else feels like it. I want access when *I* feel like it. Anyway, BuildBot is actually perfectly capable of dealing with this. I failed to separate my assumptions about how everyone would want to use the system from what the system is actually capable of. If you run a build slave and it's offline when a build is requested, the build will be queued and run when the slave comes back online. So if the CPython developers want to work this way (I wouldn't), then we don't need pony-build; BuildBot will do just fine. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
2009/10/25 : > Perhaps this is a significant portion of the problem. Maintaining a build > slave is remarkably simple and easy. I maintain about half a dozen slaves > and spend at most a few minutes a month operating them. Actually setting one > up in the first place might take a bit longer, since it involves installing > the necessary software and making sure everything's set up right, but the > actual slave configuration itself is one command: > > buildbot create-slave > > Perhaps this will help dispel the idea that it is a serious undertaking to > operate a slave. > > The real requirement which some people may find challenging is that the > slave needs to operate on a host which is actually online almost all of the > time. If you don't such a machine, then there's little point offering to > host a slave. I have been seriously considering setting up one or more buildslaves for a while now. However, my biggest issue is that they would be running as VMs on my normal PC, which means that it's the issue of keeping them continually online that hurts me. If I could (say) just fire the slaves up for a set period, or fire them up, have them do a build and report back, and then shut down, that would make my life easier (regular activities rather than ongoing sysadmin works better for me). It sounds like a buildslave isn't really what I should be looking at. Maybe Titus' push model pony-build project would make more sense for me. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Retrieve an arbitrary element from a set without removing it
On Oct 25, 2009, at 2:50 AM, Terry Reedy wrote: Alex Martelli wrote: Next(s) would seem good... That does not work. It has to be next(iter(s)), and that has been tried and eliminated because it is significantly slower. But who cares about the speed of getting an arbitrary element from a set? How can it *possibly* be a problem in a real program? If you want to optimize python, this operation is certainly not the right place to start... James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Oct 25, 2009, at 10:05 AM, exar...@twistedmatrix.com wrote: First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. Amazon's EC2 service is supported out of the box in the latest release. I have been working to expand this support to Rackspace's Cloud Servers as well. S ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
twistedmatrix.com> writes: > > To me, that raises the question of why people aren't more concerned with > the status of the build system. Shouldn't developers care if the code > they're writing works or not? The fact that we ask questions and publicly express worries should hint that we /are/ concerned :-) However, being mostly developers rather than system admins, and not knowing anything about the details of how buildbot does its work (not to mention the details of this or that particular buildslave and slave owner), makes us (at least me) quite clueless when faced with a buildbot-not-working-as-expected problem. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Oct 25, 2009, at 9:50 AM, exar...@twistedmatrix.com wrote: Actually setting one up in the first place might take a bit longer, since it involves installing the necessary software and making sure everything's set up right, but the actual slave configuration itself is one command: buildbot create-slavepassword> I have written a Fabric script for the distutils-buildbot project (on bitbucket, under Tarek) that puts everything necessary up onto an Ubuntu server, runs all the build steps, and fires up the buildbot. Obviously it will have to be modified to correctly configure other types of servers but the implementation should be fairly trivial for someone who could have done it by hand in the first place.Once it's done, it's in the script and may require an occasional tweak but not much more. The next step is to have the slaves themselves created in the cloud, fired up and then report to the mother ship that they're available. This last step is the one that doesn't seem to be supported by the current system. Thanks, S ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Oct 25, 2009, at 5:47 AM, Martin v. Löwis wrote: These are actually two issues: a) where do we get buildbot hardware and operators? I've been trying to get some feedback about firing up buildbots on Cloud Servers for a while now and haven't had much luck. I'd love to find a way of having buildbots come to life, report to the mother ship, do the build, then go away 'till next time they're required. S ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [TIP] Possible language summit topic: buildbots
On 12:48 pm, c...@msu.edu wrote: [snip] The most *exciting* part of pony-build, apart from the always-riveting spectacle of "titus rediscovering problems that buildbot solved 5 years ago", is the loose coupling of recording server to the build slaves and build reporters. My plan is to enable a simple and lightweight XML-RPC and/or REST-ish interface for querying the recording server from scripts or other Web sites. This has Brett aquiver with anticipation, I gather -- no more visual inspection of buildbot waterfall pages ;) BuildBot has an XML-RPC interface. So Brett can probably do what he wants with BuildBot right now. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Sun, Oct 25, 2009 at 8:48 AM, C. Titus Brown wrote: > [ x-posting to testing-in-python; please redirect followups to one list or > the other! ] > > Hi Mark, > > a few bits of information... > > --- > > I have a set of VM machines running some "core" build archs -- Linux, Mac OS > X, > Win XP, Win Vista, and Win 7 -- that I am going to dedicate to this purpose. > I am happy to give out remote admin access to a few people. This > infrastructure is also going to increase slowly as I build up my lab's > internal > network. > > I'm giving Tarek an account on my Linux box later today to serve as a build > slave for Distribute. Just to add to what titus said; I'm trying to price out a decent Desktop machine with enough ram/disk/cpu to run VMWare ESXi and serve a variety of virtual machines. I had planned on (ab)using the free MSDN license Microsoft provided to get a variety of platforms up and running. The end goal (since I have excess bandwidth and cooling where I live) would be to maintain this box as a series of buildslaves python-dev would have near unlimited shell/remote desktop access to. The nice thing about this would be that once the initial cost was sunk for the machine itself, and making all the virtual machines, in theory the machine could run a set of "common" virtual machines all the time, with a set of virtual machines on standby if someone needed to debug a less common problem. Yes, this is a mini-snakebite concept. Right now the main blocker is funding and time - that and I need to spec something that doesn't sound like a jet engine ;) jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On 09:47 am, mar...@v.loewis.de wrote: Mark Dickinson wrote: Would it be worth spending some time discussing the buildbot situation at the PyCon 2010 language summit? In the past, I've found the buildbots to be an incredibly valuable resource; especially when working with aspects of Python or C that tend to vary significantly from platform to platform (for me, this usually means floating-point, and platform math libraries, but there are surely many other things it applies to). But more recently there seem to have been some difficulties keeping a reasonable number of buildbots up and running. A secondary problem is that it can be awkward to debug some of the more obscure test failures on buildbots without having direct access to the machine. From conversations on IRC, I don't think I'm alone in wanting to find ways to make the buildbots more useful. These are actually two issues: a) where do we get buildbot hardware and operators? b) how can we reasonably debug problems occurring on buildbots For a), I think we can solve this only by redundancy, i.e. create more build slaves, hoping that a sufficient number would be up at any point in time. So: what specific kinds of buildbots do you think are currently lacking? A call for volunteers will likely be answered quickly. So the question is: how best to invest time and possibly money to improve the buildbot situation (and as a result, I hope, improve the quality of Python)? I don't think money will really help (I'm skeptical in general that money helps in open source projects). As for time: "buildbot scales", meaning that the buildbot slave admins will all share the load, being responsible only for their own slaves. I think that money can help in two ways in this case. First, there are now a multitude of cloud hosting providers which will operate a slave machine for you. BuildBot has even begun to support this deployment use-case by allowing you to start up and shut down vms on demand to save on costs. Amazon's EC2 service is supported out of the box in the latest release. Second, there are a number of active BuildBot developers. One of them has even recently taken a contract from Mozilla to implement some non- trivial BuildBot enhancements. I think it very likely that he would consider taking such a contract from the PSF for whatever enhancements would help out the CPython buildbot. On the master side: would you be interested in tracking slave admins? What could be done to make maintenance of build slaves easier? This is something that only the slave admins can answer. I don't think it's difficult - it's just that people are really unlikely to contribute to the same thing over a period of five years at a steady rate. So we need to make sure to find replacements when people drop out. This is a good argument for VMs. It's certainly *possible* to chase an ever changing set of platforms, but it strikes me as something of a waste of time. The source of the problem is that such a system can degrade without anybody taking action. If the web server's hard disk breaks down, people panic and look for a solution quickly. If the source control is down, somebody *will* "volunteer" to fix it. If the automated build system produces results less useful, people will worry, but not take action. To me, that raises the question of why people aren't more concerned with the status of the build system. Shouldn't developers care if the code they're writing works or not? Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On 12:16 pm, solip...@pitrou.net wrote: For a), I think we can solve this only by redundancy, i.e. create more build slaves, hoping that a sufficient number would be up at any point in time. We are already doing this, aren't we? http://www.python.org/dev/buildbot/3.x/ It doesn't seem to work very well, it's a bit like a Danaides vessel. The source of the problem is that such a system can degrade without anybody taking action. If the web server's hard disk breaks down, people panic and look for a solution quickly. If the source control is down, somebody *will* "volunteer" to fix it. If the automated build system produces results less useful, people will worry, but not take action. Well, to be fair, buildbots breaking also happens much more frequently (perhaps one or two orders of magnitude) than the SVN server or the Web site going down. Maintaining them looks like a Sisyphean task, and nobody wants that. Perhaps this is a significant portion of the problem. Maintaining a build slave is remarkably simple and easy. I maintain about half a dozen slaves and spend at most a few minutes a month operating them. Actually setting one up in the first place might take a bit longer, since it involves installing the necessary software and making sure everything's set up right, but the actual slave configuration itself is one command: buildbot create-slavepassword> Perhaps this will help dispel the idea that it is a serious undertaking to operate a slave. The real requirement which some people may find challenging is that the slave needs to operate on a host which is actually online almost all of the time. If you don't such a machine, then there's little point offering to host a slave. I don't know what kind of machines are the current slaves, but if they are 24/7 servers, isn't it a bit surprising that the slaves would go down so often? Is the buildbot software fragile? Does it require a lot of (maintenance, repair) work from the slave owners? As I have no specific experience maintaining any of the CPython build slaves, I can't speak to any maintenance issues which these slaves have encountered. I would expect that they are as minimal as the issues I have encountered maintaining slaves for other projects, but perhaps this is wrong. I do recall that there were some win32 issues (discussed on this list, I think) quite a while back, but I think those were resolved. I haven't heard of any other issues since then. If there are some, perhaps the people who know about them could raise them and we could try to figure out how to resolve them. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
On Sun, Oct 25, 2009 at 08:54:46AM +, Mark Dickinson wrote: > Would it be worth spending some time discussing the buildbot situation > at the PyCon 2010 language summit? In the past, I've found the > buildbots to be an incredibly valuable resource; especially when > working with aspects of Python or C that tend to vary significantly > from platform to platform (for me, this usually means floating-point, > and platform math libraries, but there are surely many other things it > applies to). But more recently there seem to have been some > difficulties keeping a reasonable number of buildbots up and running. > A secondary problem is that it can be awkward to debug some of the > more obscure test failures on buildbots without having direct access > to the machine. From conversations on IRC, I don't think I'm alone in > wanting to find ways to make the buildbots more useful. > > So the question is: how best to invest time and possibly money to > improve the buildbot situation (and as a result, I hope, improve the > quality of Python)? What could be done to make maintenance of build > slaves easier? Or to encourage interested third parties to donate > hardware and time? Are there good alternatives to Buildbot that might > make a difference? What do other projects do? > > These are probably the wrong questions; I'm hoping that a discussion > would help produce the right questions, and possibly some answers. [ x-posting to testing-in-python; please redirect followups to one list or the other! ] Hi Mark, a few bits of information... --- I have a set of VM machines running some "core" build archs -- Linux, Mac OS X, Win XP, Win Vista, and Win 7 -- that I am going to dedicate to this purpose. I am happy to give out remote admin access to a few people. This infrastructure is also going to increase slowly as I build up my lab's internal network. I'm giving Tarek an account on my Linux box later today to serve as a build slave for Distribute. -- More machines, and more esoteric machines, will be coming online as Snakebite unfolds. Trent Nelson (Snakepit master) has been finishing up some consulting work and is going to dedicate his time to it starting in November. This means more 64 bit, bigmem, and "weird" archs, with full login access. --- I've also been working on a buildbot alternative that I'm calling pony-build. pony-build is based on a client-push architecture in which client machines do builds and push results to the master, which then acts as a record-keeper rather than a slave driver. The result is a less coordinated but (AFAICT) much less fragile continuous integration system. I'm hoping to give a talk at PyCon on the differences, and there will be a sprint on pony-build + pyhton-dev at PyCon, regardless. The current status of pony-build is "functional but ugly inside". In particular, the data model is horrible, and the internal API needs much more fleshing out. Nonetheless, my server has been running for two months or so, and you can look at the results here, http://lyorn.idyll.org/ctb/pb-dev/ The most fully-fleshed out set of build clients is for pygr, http://lyorn.idyll.org/ctb/pb-dev/pygr/ but you can view daily build results for Python 2.7 at http://lyorn.idyll.org/ctb/pb-dev/python/ with an exhaustive list here http://lyorn.idyll.org/ctb/pb-dev/python/show_all (and why the heck am I sorting in reverse date order, anyway?!) The most interesting (?) part of pony-build right now is the client config, which I'm struggling to make simple and potentially universal enough to serve under buildbot as well: http://github.com/ctb/pony-build/blob/master/client/build-cpython (see 'commands' list). The most *exciting* part of pony-build, apart from the always-riveting spectacle of "titus rediscovering problems that buildbot solved 5 years ago", is the loose coupling of recording server to the build slaves and build reporters. My plan is to enable a simple and lightweight XML-RPC and/or REST-ish interface for querying the recording server from scripts or other Web sites. This has Brett aquiver with anticipation, I gather -- no more visual inspection of buildbot waterfall pages ;) -- If you're interested in bashing on, contributing to, or helping figure out what color the pony-build main screen should be, contact me off-list; I'm reluctant to spam up python-dev or testing-in-python. That having been said, the results of taking it and trying it out -- you can post results to my own recording server at http://lyorn.idyll.org/ctb/pb-dev/xmlrpc -- would be most welcome. Once I fix the data model, code collaboration will be much more feasible, too. --- cheers, --titus -- C. Titus Brown, c...@msu.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-a
Re: [Python-Dev] Possible language summit topic: buildbots
> For a), I think we can solve this only by redundancy, i.e. create more > build slaves, hoping that a sufficient number would be up at any point > in time. We are already doing this, aren't we? http://www.python.org/dev/buildbot/3.x/ It doesn't seem to work very well, it's a bit like a Danaides vessel. > The source of the problem is that such a system can degrade without > anybody taking action. If the web server's hard disk breaks down, people > panic and look for a solution quickly. If the source control is down, > somebody *will* "volunteer" to fix it. If the automated build system > produces results less useful, people will worry, but not take action. Well, to be fair, buildbots breaking also happens much more frequently (perhaps one or two orders of magnitude) than the SVN server or the Web site going down. Maintaining them looks like a Sisyphean task, and nobody wants that. I don't know what kind of machines are the current slaves, but if they are 24/7 servers, isn't it a bit surprising that the slaves would go down so often? Is the buildbot software fragile? Does it require a lot of (maintenance, repair) work from the slave owners? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Clean up Python/thread_*.h ?
Martin v. Löwis wrote: Making a decision and removing the files considered unnecessary would clarify what platforms still are/should be supported. I think any such removal should go through the PEP 11 process. Put a #error into the files for 3.2, and a removal notice into the PEP, then remove them in 3.3. As for OS/2: this should probably stay. Yes please (to keeping the OS/2 thread support as-is). Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible language summit topic: buildbots
Mark Dickinson wrote: > Would it be worth spending some time discussing the buildbot situation > at the PyCon 2010 language summit? In the past, I've found the > buildbots to be an incredibly valuable resource; especially when > working with aspects of Python or C that tend to vary significantly > from platform to platform (for me, this usually means floating-point, > and platform math libraries, but there are surely many other things it > applies to). But more recently there seem to have been some > difficulties keeping a reasonable number of buildbots up and running. > A secondary problem is that it can be awkward to debug some of the > more obscure test failures on buildbots without having direct access > to the machine. From conversations on IRC, I don't think I'm alone in > wanting to find ways to make the buildbots more useful. These are actually two issues: a) where do we get buildbot hardware and operators? b) how can we reasonably debug problems occurring on buildbots For a), I think we can solve this only by redundancy, i.e. create more build slaves, hoping that a sufficient number would be up at any point in time. So: what specific kinds of buildbots do you think are currently lacking? A call for volunteers will likely be answered quickly. > So the question is: how best to invest time and possibly money to > improve the buildbot situation (and as a result, I hope, improve the > quality of Python)? I don't think money will really help (I'm skeptical in general that money helps in open source projects). As for time: "buildbot scales", meaning that the buildbot slave admins will all share the load, being responsible only for their own slaves. On the master side: would you be interested in tracking slave admins? > What could be done to make maintenance of build > slaves easier? This is something that only the slave admins can answer. I don't think it's difficult - it's just that people are really unlikely to contribute to the same thing over a period of five years at a steady rate. So we need to make sure to find replacements when people drop out. > Or to encourage interested third parties to donate > hardware and time? Again: I think a call for volunteers would do (Steve, if you are reading this, please hold back just a few days before actually making such a call :-) > Are there good alternatives to Buildbot that might > make a difference? I think people have started working on such a thing. There are certainly alternatives; I'm fairly skeptical that they are *good* alternatives (but then, I'm the one who set up the buildbot installation in the first place). > What do other projects do? I think that's really difficult to compare, since their testing often has a very different scope. I think CruiseControl is widely used. > These are probably the wrong questions; I'm hoping that a discussion > would help produce the right questions, and possibly some answers. I think these are good questions - just not for the summit. Setting up such a system is, conceptually, easy. It's also just a little work to set it up initially; the difficult part then is to keep it running (and no, a system where anybody can just post test results at any time without prior registration is *still* difficult to keep running). The source of the problem is that such a system can degrade without anybody taking action. If the web server's hard disk breaks down, people panic and look for a solution quickly. If the source control is down, somebody *will* "volunteer" to fix it. If the automated build system produces results less useful, people will worry, but not take action. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Possible language summit topic: buildbots
Would it be worth spending some time discussing the buildbot situation at the PyCon 2010 language summit? In the past, I've found the buildbots to be an incredibly valuable resource; especially when working with aspects of Python or C that tend to vary significantly from platform to platform (for me, this usually means floating-point, and platform math libraries, but there are surely many other things it applies to). But more recently there seem to have been some difficulties keeping a reasonable number of buildbots up and running. A secondary problem is that it can be awkward to debug some of the more obscure test failures on buildbots without having direct access to the machine. From conversations on IRC, I don't think I'm alone in wanting to find ways to make the buildbots more useful. So the question is: how best to invest time and possibly money to improve the buildbot situation (and as a result, I hope, improve the quality of Python)? What could be done to make maintenance of build slaves easier? Or to encourage interested third parties to donate hardware and time? Are there good alternatives to Buildbot that might make a difference? What do other projects do? These are probably the wrong questions; I'm hoping that a discussion would help produce the right questions, and possibly some answers. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
