Re: [Python-Dev] Buildbot questions
Ah, but that would require changes to the slaves, right? I would prefer a solution that avoids that. I don't think so. In my little test setup I didn't have to make any change to the slave. The update and clobber mode parameters are implemented on the slave side. Trent's patch changes which mode setting is used on the master side before anything is sent to the slaves, so the slaves don't have to know anything about it. In general there is as little slave-side code as possible, to reduce the need to upgrade or change the code on that side. The VC checkout/update operations are an exception, since they may do several operations in a row (which would mean a lot more latency to do them without support from code on the slave side), and because I'm lazy and prefer to implement the algorithms in Python instead of in Bourne shell :). -Brian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
On Friday 06 January 2006 18:39, Martin v. Löwis wrote: I would like to do this in buildbot, but I'm not sure how to (i.e. wipe the build occasionally, but not every time). For example, I could imagine completely cleaning the build directory every time the build number % 10 == 0. Still, what the precise buildbot incantation would be, I don't know. At least with the way Twisted is set up, the buildbot also sits in an IRC channel and sends updates there. It can also be controlled from there. Is this worth doing? A 'force clean build' command could be added... Anthony -- Anthony Baxter [EMAIL PROTECTED] It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
[Martin v. Loewis wrote] I would like to do this in buildbot, but I'm not sure how to (i.e. wipe the build occasionally, but not every time). For example, I could imagine completely cleaning the build directory every time the build number % 10 == 0. Still, what the precise buildbot incantation would be, I don't know. (Still learning my buildbot mojo.) One idea would be to do what Mozilla's Tinderbox does: they have one set of builds that are incremental and one set that are full. Actually looking around on tinderbox.mozilla.org I could only find incremental builds so I'm not sure what they are doing. To wipe out the build occassionally you could (presumably) add a starting step to the Python 'builder' (in the build master.cfg) to rm -rf $builddir every, say, Sunday night. Trent -- Trent Mick [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
Trent Mick wrote: (Still learning my buildbot mojo.) One idea would be to do what Mozilla's Tinderbox does: they have one set of builds that are incremental and one set that are full. Actually looking around on tinderbox.mozilla.org I could only find incremental builds so I'm not sure what they are doing. I know how to set up full vs. incremental *builders*; they would appear in different lanes in the Waterfall, and they would operate on different build directories. I don't think I want additional lanes to the Waterfall. To wipe out the build occassionally you could (presumably) add a starting step to the Python 'builder' (in the build master.cfg) to rm -rf $builddir every, say, Sunday night. Sure, that would be the idea. How to formulate it? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
Anthony Baxter wrote: At least with the way Twisted is set up, the buildbot also sits in an IRC channel and sends updates there. It can also be controlled from there. Is this worth doing? A 'force clean build' command could be added... The problem with irc-enabling (or web-enabling) such things is that there is a potential for abuse. Of course, in this case, we could wait for the abuse to happen. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
To wipe out the build occassionally you could (presumably) add a
starting step to the Python 'builder' (in the build master.cfg) to
rm -rf $builddir
every, say, Sunday night.
Sure, that would be the idea. How to formulate it?
I think I'm part of the way there with the following. I've subclassed
the SVN source build step to add support for new source mode:
update_and_clobber_occassionally. Basically it (hackily) changes the
source type btwn update, which we usually want, and clobber, which
we sometimes want.
The current problem with this is that I don't know if the build step
objects have access to a local data store -- so it could, say, count how
many builds have been done to know to clobber every tenth one. The
current code just chooses to clobber for any build on Sunday. Not a very
satisfying solution.
Brian,
Is there a way for build steps to maintain some state?
clipped from the build master's master.cfg -
...
class SVNEx(step.SVN):
A SVN Source build step that adds the ability for the Source
mode to be a combination of update and clobber. This is useful if
your general just want to update the source tree from source code
control,
but occassionally want to clobber and start fresh.
To use this functionality use mode=update_and_clobber_occassionally.
To control when occassionally is now override the should_clobber
method. The default implementation is currently hardcoded to
every Sunday. (This is a HACK. Perhaps there should be constructor
options to clobber every Nth time or to have cron-like arguments -- see
Nightly in scheduler.py. I don't know if steps have access to a
local
data store to be able to do this -- e.g. keep a counter.)
Ideally this would be an option of the base Source class in
buildbot/process/step.py.
def __init__(self, **kwargs):
if kwargs.has_key(mode) \
and kwargs[mode] == update_and_clobber_occassionally:
self.update_and_clobber_occassionally = True
kwargs[mode] = update
else:
self.update_and_clobber_occassionally = False
step.SVN.__init__(self, **kwargs)
def should_clobber(self):
from datetime import date
is_sunday = date.today().weekday() == 6 # it is Sunday
from twisted.python import log
log.msg(should_clobber? %s, (is_sunday and yes or no))
return is_sunday
def startVC(self, *args, **kwargs):
if self.update_and_clobber_occassionally:
if self.should_clobber():
self.args[mode] = clobber
else:
self.args[mode] = update
step.SVN.startVC(self, *args, **kwargs)
python_factory = factory.GNUAutoconf(
s(SVNEx, svnurl=http://svn.python.org/projects/python/trunk;,
mode=update_and_clobber_occassionally),
test=[make, test], # use `make testall`?
)
# Then use python_factory as something like so:
#c['builders'] = [
#{'name': linux-x86,
# 'slavename': ...,
# 'builddir': python,
# 'factory': python_factory,
# },
#...
#]
clipped from the build master's master.cfg -
Cheers,
Trent
--
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
Trent Mick wrote: I think I'm part of the way there with the following. I've subclassed the SVN source build step to add support for new source mode: update_and_clobber_occassionally. Basically it (hackily) changes the source type btwn update, which we usually want, and clobber, which we sometimes want. Ah, but that would require changes to the slaves, right? I would prefer a solution that avoids that. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
[Martin v. Loewis wrote] Trent Mick wrote: I think I'm part of the way there with the following. I've subclassed the SVN source build step to add support for new source mode: update_and_clobber_occassionally. Basically it (hackily) changes the source type btwn update, which we usually want, and clobber, which we sometimes want. Ah, but that would require changes to the slaves, right? I would prefer a solution that avoids that. I don't think so. In my little test setup I didn't have to make any change to the slave. Trent -- Trent Mick [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
On Saturday 07 January 2006 10:01, Martin v. Löwis wrote: The problem with irc-enabling (or web-enabling) such things is that there is a potential for abuse. Of course, in this case, we could wait for the abuse to happen. That would be my vote. Worst comes to worst, we lock it down to a list of known users. Anthony -- Anthony Baxter [EMAIL PROTECTED] It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Buildbot questions
I currently have a (quite weak) computer that mostly sits idle (shares the web connection), Tbird 750; 640Mb RAM; Windows Server 2003 Standard Edition. Since the computer sits there doing nothing, I could probably put a buildbot on it if needed (since the python-dev thread states that many windows flavour would be appreciable and that Win2003 may not be extremely common), but i'd like to know how often it'll try to build, and how long the build itself may take on such a platform. Morel Xavier ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
Morel but i'd like to know how often it'll try to build, and how long Morel the build itself may take on such a platform. It should build every time there's a checkin on trunk or the 2.4 branch. As for performance, take a look at http://www.python.org/dev/buildbot/ to see how long some of the other build bots take and extrapolate from that. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
FWIW, I have an older box running Ubuntu 05.10 that spends most of it's days pining for stuff to do (at the moment, it does DHCP and DNS for the house. Yes, I know I have too many damn computers here). I can set up a buildbot on it easily enough. It's something like a 600MHz P3 or something. Is it going to be useful to have this in the mix? Heck, there's a pile of 500MHz P3s sitting here that I could drop a random free unix onto if someone wants to nominate something that's a) useful b) not a total pain in the clacker to install. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
[Morel Xavier] I currently have a (quite weak) computer that mostly sits idle (shares the web connection), Tbird 750; 640Mb RAM; Windows Server 2003 Standard Edition. Since the computer sits there doing nothing, I could probably put a buildbot on it if needed (since the python-dev thread states that many windows flavour would be appreciable and that Win2003 may not be extremely common), but i'd like to know how often it'll try to build, and how long the build itself may take on such a platform. A problem for Windows buildbot slaves is that they need an appropriate compiler. Does this machine have MS VC 7.1 installed? If not, it can't compile the code. The Windows Python would also like to build several other packages (like bz2 and Tcl/Tk). An update style of slave does an svn update rather than a full checkout, and that usually goes very fast after the first time. Likewise compiling if binaries are left behind across runs. For the rest, open a DOS box on this machine, cd to root of a Python 2.4.2 installation, and time python Lib\test\regrtest.py -uall That's about how long it will take on that machine to run all the tests from the current trunk too. Really thorough tests take 8x as long (with and without purging .pyc/.pyo files first, with and without -O, and under release and debug builds: 2*2*2 = 8). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
On Thu, 5 Jan 2006, Tim Peters wrote: [...] A problem for Windows buildbot slaves is that they need an appropriate compiler. Does this machine have MS VC 7.1 installed? If not, it can't compile the code. The Windows Python would also like to build several other packages (like bz2 and Tcl/Tk). Might a buildbot running this setup of David Munman's (free MS compiler + NAnt interpreting the MS project file) be useful? http://groups.google.co.uk/group/comp.lang.python/browse_frm/thread/226584dd47047bb6/1e33ad19197bee20 (I'm not offering a buildbot myself, just pointing out the possibility of using this tool) There always the chance it gets something wrong or falls over while trying to interpret the project file, of course. That in itself would be beneficial, though, if there's a committer willing to fix it when it breaks -- I currently have access to MS compilers, but I remember many happy :-/ hours as a graduate student trying to compile Fortran and C extensions on win32 with free compilers. An update style of slave does an svn update rather than a full checkout, and that usually goes very fast after the first time. Likewise compiling if binaries are left behind across runs. [...] Much though I like SVN, it seems its working copy management still leaves a little to be desired: Even quite recently (fairly sure it was client version 1.2.*, on Win XP) and with read-only checkouts used only for builds, I've still seen it end up in an incorrect state. I suspect 'svn switch' or 'svn up -r x' was the culprit, though, so perhaps it's not a problem if exactly 'svn up' is the only svn command ever executed on the checkout. Still, perhaps it's wise to wipe the checkout every so often? John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
Anthony Baxter wrote: FWIW, I have an older box running Ubuntu 05.10 that spends most of it's days pining for stuff to do (at the moment, it does DHCP and DNS for the house. Yes, I know I have too many damn computers here). I can set up a buildbot on it easily enough. It's something like a 600MHz P3 or something. Is it going to be useful to have this in the mix? With the gentoo installation, I think we have enough linux for the moment. Somebody noticed that the Waterfall view of buildbot quickly becomes unreadable if there are too many builds. Heck, there's a pile of 500MHz P3s sitting here that I could drop a random free unix onto if someone wants to nominate something that's a) useful b) not a total pain in the clacker to install. For a), I think one of the BSDs might be useful. Whether they qualify for b), I don't know. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
John J Lee wrote: Might a buildbot running this setup of David Munman's (free MS compiler + NAnt interpreting the MS project file) be useful? I feel that any contribution here takes quite some time initially, so somebody making that offer should accept some pain until it really works self-contained. I would need to know the exact sequence of commands that are necessary for the build; I assume the autoconf builder would be useless. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
On Friday 06 January 2006 07:44, Martin v. Löwis wrote: With the gentoo installation, I think we have enough linux for the moment. Somebody noticed that the Waterfall view of buildbot quickly becomes unreadable if there are too many builds. My only concern is that it's gentoo, not just linux. I know that for a couple of my other open source projects I usually don't spend too long debugging bizarrely broken apparent bugs, because it ends up being some strange build flag or some such on the gentoo box in question. On the other hand, this box is unlikely to have been built with a selection of gcc flags that Neal just selected randomly from the gcc manual wink so it's probably going to be better than that. Heck, there's a pile of 500MHz P3s sitting here that I could drop a random free unix onto if someone wants to nominate something that's a) useful b) not a total pain in the clacker to install. For a), I think one of the BSDs might be useful. Whether they qualify for b), I don't know. Anyone else have an opinion on the ease of installation for the various BSDs? Last time I tried one (which was several years ago) it was Not Very Good. Anthony -- Anthony Baxter [EMAIL PROTECTED] It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
On Jan 5, 2006, at 1:08 PM, Anthony Baxter wrote: On Friday 06 January 2006 07:44, Martin v. Löwis wrote: With the gentoo installation, I think we have enough linux for the moment. Somebody noticed that the Waterfall view of buildbot quickly becomes unreadable if there are too many builds. My only concern is that it's gentoo, not just linux. I know that for a couple of my other open source projects I usually don't spend too long debugging bizarrely broken apparent bugs, because it ends up being some strange build flag or some such on the gentoo box in question. On the other hand, this box is unlikely to have been built with a selection of gcc flags that Neal just selected randomly from the gcc manual wink so it's probably going to be better than that. Heck, there's a pile of 500MHz P3s sitting here that I could drop a random free unix onto if someone wants to nominate something that's a) useful b) not a total pain in the clacker to install. For a), I think one of the BSDs might be useful. Whether they qualify for b), I don't know. Anyone else have an opinion on the ease of installation for the various BSDs? Last time I tried one (which was several years ago) it was Not Very Good. FreeBSD and OpenBSD are painless these days, assuming that you're comfortable reading text during installation. No experience with NetBSD. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
On Thu, 5 Jan 2006, John J Lee wrote: Might a buildbot running this setup of David Munman's (free MS compiler + NAnt interpreting the MS project file) be useful? s/Munman/Murmann/ John ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
Anthony Baxter wrote: My only concern is that it's gentoo, not just linux. I know that for a couple of my other open source projects I usually don't spend too long debugging bizarrely broken apparent bugs, because it ends up being some strange build flag or some such on the gentoo box in question. For buildbot, failures to build are good (if they are reproducable); if it succeeds for gentoo, but fails on some other system on which it ought to work, then adding this system as a build slave would be useful. Regards, Martin P.S. That's assuming we get in a state where the tests usually pass, of course. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot questions
[John J Lee] Might a buildbot running this setup of David Munman's (free MS compiler + NAnt interpreting the MS project file) be useful? http://groups.google.co.uk/group/comp.lang.python/browse_frm/thread/226584dd47047bb6/1e33ad19197bee20 No comment from me about that (don't know anything about it, and don't have time to learn). Someone who wants to see a platform exercised needs to volunteer that platform themself (and realize that every stinkin' little step has to be so well defined that the people running the buildbot _master_ can send the exact steps needed to the slave). ... Much though I like SVN, it seems its working copy management still leaves a little to be desired: Even quite recently (fairly sure it was client version 1.2.*, on Win XP) and with read-only checkouts used only for builds, I've still seen it end up in an incorrect state. I suspect 'svn switch' or 'svn up -r x' was the culprit, though, so perhaps it's not a problem if exactly 'svn up' is the only svn command ever executed on the checkout. I doubt anyone slings more svn projects, branches and tags than Zope Corp, and I've had no problems with svn on WinXP there except when a project switches from making copies of externals to getting them via svn:externals instead -- and then everyone has problems, regardless of platform. What I _have_ had problems with is PuTTY, and recently discovered that all my months-long svn+ssh problems went away after backing off to the older PuTTY 0.57 (and come back again immediately upon switching to 0.58). Still, perhaps it's wise to wipe the checkout every so often? I think it is. And while I haven't seen this under MS VC7.1 yet, a few times I caught VC 6.0 failing to recompile after a relevant header file changed. Certainly from-scratch checkout + build should be done before a release. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
