Re: [Python-Dev] AIX 5.3 - Enabling Shared Library Support Vs Extensions
Hi Anurag, Le 25/11/2010 10:24, Anurag Chourasia a écrit : All, When I configure python to enable shared libraries, none of the extensions are getting built during the make step due to this error. you may want to take a look at the following issue: http://bugs.python.org/issue941346 Python compiled with shared libraries was broken on AIX until recently. There are some patches there to get it to work, or you may want to test the latest 2.7 or 3.x releases. regards -- Sébastien Sablé ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot for AIX
Hi Antoine, I tried to provide command lines arguments to configure instead of environment variables with: configureFlags = [--with-pydebug, --without-computed-gotos, CC=xlc, 'OPT=-O2 -qmaxmem=18000'] But that would fail: on the slave, configure would run like that: ./configure --with-pydebug --without-computed-gotos CC=xlc OPT=-O2 -qmaxmem=18000 And the compilation would give some error like that: xlc -c -O2 -qmaxmem=18000 -O2 -O2 -I. -IInclude -I./Include -I/home/cis/.buildbot/support-buildbot/include -I/home/cis/.buildbot/support-buildbot/include/ncurses -I/home/cis/.buildbot/support-buildbot/include -I/home/cis/.buildbot/support-buildbot/include/ncurses -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c xlc: 1501-216 (W) command option - -qmaxmem=18000 is not recognized - passed to ld However running 2 different slaves per host in order to distinguish xlc and gcc would be OK; though I would appreciate if they could run sequentially rather than in parallel as that would limit the host load. regards -- Sébastien Sablé Le 28/10/2010 16:45, Antoine Pitrou a écrit : On Fri, 15 Oct 2010 17:38:47 +0200 Sébastien Sablésa...@users.sourceforge.net wrote: Could you please take a look at those modifications in master.cfg, provide me some password for the bot slaves and apply the corrections in those issues? About the master.cfg modifications: there should be no need for separate environment variables. Instead, you should be able to specify them as command-line arguments to ./configure, e.g.: [--with-pydebug, --without-computed-gotos, CC=xlc, 'OPT=-O2 -qmaxmem=18000'] Can you check this works for you? Also, there's no need to complicate the buildbot naming procedure. You should be able to run several buildslaves on a single machine, provided we give you separate credentials: one per compiler type. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/sable%40users.sourceforge.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot for AIX
Hi Martin, I got no reply concerning those modifications to the buildbot script so that I could run 2 slaves on AIX. I am able to spend some time at the moment on Python for AIX because we are migrating the product in my company from AIX 5.3 to AIX 6.1. However once this is done, by the end of November, it will be more difficult for me to justify spending too much time on this builbot installation and on Python AIX bugs. So it would be nice if I could get this slave running as soon as possible, so that I could spend a few more days solving some of the remaining issues on this platform. regards -- Sébastien Sablé Le 15/10/2010 17:38, Sébastien Sablé a écrit : Hi Martin, I finally got the authorization to run some buildbot slaves on our AIX servers. I made some tests with a buildbot script as close as possible to what you already use. Here is a patch that show the kind of modifications I had to do in order to get the buildbot slave to correctly run on AIX. It is also necessary to apply the patch provided in issue 9862 so that the tests won't hang forever. It also would help if the corrections provided in the following issues could be commited: 4499, 678250, 730467. Could you please take a look at those modifications in master.cfg, provide me some password for the bot slaves and apply the corrections in those issues? Once this is done, I can run 2 buildbot slaves for AIX 6.1 and AIX 5.3 with build plans for gcc and xlc. I can't guarantee that those bots will run forever, and I may have to ask to schedule them only at night if the activity it generates on the server appears to be too high. But I think it could greatly help to improve the state of Python on AIX (which is far from perfect at the moment). regards -- Sébastien Sablé Le 20/09/2010 21:21, Martin v. Löwis a écrit : Also could you provide me the master.cfg file (with obfuscated passwords) that is used by the Python buildbot master or tell me if it is in subversion somewhere? Attached! Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot for AIX
Hi Sridhar, Le 15/10/2010 18:50, Sridhar Ratnakumar a écrit : We definitely like to share our core Python patches for AIX 5.1/5.2 and other platforms. Great to hear that ActiveState shares their improvements for Python on AIX! Thanks for the patch in git, I will check it and try to open some issues in the Python bug tracker in order to get those modifications integrated. Please do send us any patches you may have for building Python 3 on AIX. Here is a list of issues concerning AIX on which I have been working: http://bugs.python.org/issue678250 = OK (test_mmap error) http://bugs.python.org/issue678264 = WAIT (test_resource error) http://bugs.python.org/issue730467 = WAIT (AIX_GENUINE_CPLUSPLUS error) http://bugs.python.org/issue713169 (test_pty error) http://bugs.python.org/issue941346 = OK (shared library) http://bugs.python.org/issue1542544 = OK (shared library - duplicate) http://bugs.python.org/issue1563807 (compilation ctypes) http://bugs.python.org/issue1633863 = OK (bad default compiler) http://bugs.python.org/issue1745108 (curses error) http://bugs.python.org/issue3526 (custom memory allocation system - dlmalloc) http://bugs.python.org/issue4026 = OK (fcntl extension) http://bugs.python.org/issue4499 = OK (tilde macro) http://bugs.python.org/issue5718 (ctypes compilation) http://bugs.python.org/issue6006 (ctypes compilation) http://bugs.python.org/issue6164 (flag -blibpath) http://bugs.python.org/issue7657 (test_ctypes error) http://bugs.python.org/issue8695 = OK (crash on Python 2.6.4 at compilation) http://bugs.python.org/issue8882 (pb getsockaddrarg) http://bugs.python.org/issue9700 = OK (semwait) http://bugs.python.org/issue9799 = (xlc requires --without-computed-gotos) = OK add some doc http://bugs.python.org/issue9862 = OK (broken PIPE_BUF on AIX) The 2 most interesting patches I think are: * in issue 941346, a patch to be able to compile Python as a shared library (already applied in svn) * in issue 3526, a patch to use a custom memory allocation system (dlmalloc) in Python in order to greatly reduce memory consumption on AIX = not applied yet; I need to formally discuss it on Python-dev, but I am waiting for the AIX buildbot to be setup before that. best regards -- Sébastien Sablé ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot for AIX
Hi Martin,
I finally got the authorization to run some buildbot slaves on our AIX
servers.
I made some tests with a buildbot script as close as possible to what
you already use. Here is a patch that show the kind of modifications I
had to do in order to get the buildbot slave to correctly run on AIX.
It is also necessary to apply the patch provided in issue 9862 so that
the tests won't hang forever.
It also would help if the corrections provided in the following issues
could be commited: 4499, 678250, 730467.
Could you please take a look at those modifications in master.cfg,
provide me some password for the bot slaves and apply the corrections in
those issues?
Once this is done, I can run 2 buildbot slaves for AIX 6.1 and AIX 5.3
with build plans for gcc and xlc.
I can't guarantee that those bots will run forever, and I may have to
ask to schedule them only at night if the activity it generates on the
server appears to be too high.
But I think it could greatly help to improve the state of Python on AIX
(which is far from perfect at the moment).
regards
--
Sébastien Sablé
Le 20/09/2010 21:21, Martin v. Löwis a écrit :
Also could you provide me the master.cfg file (with obfuscated
passwords) that is used by the Python buildbot master or tell me if it
is in subversion somewhere?
Attached!
Regards,
Martin
--- master.cfg.orig 2010-10-15 15:38:47.0 +0200
+++ master.cfg.new 2010-10-15 16:58:54.0 +0200
@@ -27,7 +27,8 @@
# tuple of bot-name and bot-password. These correspond to values given to the
# buildslave's mktap invocation.
c['slaves'] = [
-XXX
+BuildSlave(sable-aix61, bot1passwd, max_builds=1),
+BuildSlave(sable-aix53, bot2passwd, max_builds=1),
]
@@ -80,6 +81,7 @@
class UnixBuild(factory.GNUAutoconf):
configureFlags = [--with-pydebug, --with-computed-gotos]
+configureEnv = {}
def __init__(self, source, parallel):
compile = ['make', 'all']
test = [make, buildbottest]
@@ -88,6 +90,7 @@
test = [make, buildbottest, TESTOPTS=+parallel]
factory.GNUAutoconf.__init__(self, source,
configureFlags = self.configureFlags,
+ configureEnv = self.configureEnv,
compile=compile,
test = None)
# XXX(nnorwitz): reduced timeout, no test should take longer than 10m
@@ -99,6 +102,21 @@
class WideUnixBuild(UnixBuild):
configureFlags = [--with-pydebug, --with-computed-gotos, --with-wide-unicode]
+class Aix5GccUnixBuild(UnixBuild):
+configureFlags = [--with-pydebug, --with-computed-gotos, --disable-ipv6, --with-gcc]
+
+class Aix5XlcUnixBuild(UnixBuild):
+configureFlags = [--with-pydebug, --without-computed-gotos, --disable-ipv6]
+configureEnv = {'CC': 'xlc', 'OPT': -O2 -qmaxmem=18000}
+
+class Aix6GccUnixBuild(UnixBuild):
+configureFlags = [--with-pydebug, --with-computed-gotos, --with-gcc]
+configureEnv = {'CC': 'gcc'}
+
+class Aix6XlcUnixBuild(UnixBuild):
+configureFlags = [--with-pydebug, --without-computed-gotos]
+configureEnv = {'CC': 'xlc', 'OPT': -O2 -qmaxmem=18000}
+
class CygwinBuild(UnixBuild):
def __init__(self, source, parallel):
UnixBuild.__init__(self, source)
@@ -175,35 +193,39 @@
STABLE=.stable
UNSTABLE=.unstable
-builders = [(sparc solaris10 gcc, loewis-sun, UnixBuild,STABLE),
-(amd64 gentoo, norwitz-amd64, UnixBuild,STABLE),
-(x86 gentoo, norwitz-x86, UnixBuild,UNSTABLE),
-#(g4 osx.4, psf-g4, UnixBuild,STABLE),
-#(alpha Tru64 5.1, norwitz-tru64, UnixBuild,UNSTABLE),
-(i386 Ubuntu, klose-ubuntu-i386, UnixBuild,UNSTABLE),
-(ia64 Ubuntu, klose-debian-ia64, UnixBuild,STABLE),
-(sparc Debian, klose-debian-sparc, UnixBuild,STABLE),
-(alpha Debian, klose-debian-alpha, UnixBuild,UNSTABLE),
-#(MIPS Debian, klose-debian-mips, UnixBuild,UNSTABLE),
-#(PPC64 Debian, klose-debian-ppc64, UnixBuild,UNSTABLE),
-#(S-390 Debian, klose-debian-s390, UnixBuild,UNSTABLE),
-#(x86 mvlgcc, loewis-linux, UnixBuild,UNSTABLE),
-(x86 XP-4, bolen-windows, WindowsBuild,STABLE),
-(x86 FreeBSD, bolen-freebsd, UnixBuild,UNSTABLE),
-(x86 Windows7, bolen-windows7, WindowsBuild,UNSTABLE),
-(x86 FreeBSD 7.2, bolen-freebsd7, UnixBuild,UNSTABLE),
-(x86 Ubuntu, bolen-ubuntu, UnixBuild,UNSTABLE),
- (ARMv7Thumb Ubuntu, klose-linux-arm, UnixBuild,UNSTABLE),
- (ARMv4 Debian, klose-linux-armeabi, UnixBuild,UNSTABLE),
- #(x86 FreeBSD 2, werven-freebsd, UnixBuild,UNSTABLE),
-#(x86 Gentoo, murray-gentoo, UnixBuild, UNSTABLE),
-#(x86 Gentoo wide, murray-gentoo-wide, WideUnixBuild, UNSTABLE),
-(x86 XP-5, moore-windows, WindowsBuild, UNSTABLE),
-(x86 Tiger, bolen-tiger
[Python-Dev] Buildbot for AIX
Le 17/09/2010 15:05, Antoine Pitrou a écrit : Following on Martin's comments, you might also want to share things with the ActiveState guys who, AFAIK, maintain an AIX version of Python (but you have been the most active AIX user on the bug tracker lately; perhaps they are keeping their patches to themselves). I tried to find some source package or subversion repository on their web site, but they only distribute binary versions. Also it is only the Business Edition that supports AIX and it is clearly sold with a proprietary license. So I doubt they would share their patches related to AIX, but I can always ask. Regards -- Sébastien Sablé ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Buildbot for AIX
Hi Martin, Le 17/09/2010 14:42, Martin v. Löwis a écrit : If you are having the build slave compile Python, I'd like to point out that you *already* run arbitrary shell commands provided by some external source: if somebody would check some commands into Python's configure.in, you would unconditionally execute them. So if it's ok that you run the Python build process at all, it should (IMO) also be acceptable to run a build slave. If there are concerns that running it under your Unix account gives it too much power, you should create a separate, locked-down account. Someone messing with the configure script in python svn would probably get noticed very quickly, but I agree this is also a security risk, and the buildbot slave runs with a user with limited privileges. I will try to convince the IT Team that this is an acceptable risk and setup a chroot or something like that for the buildbot slave. That may take some time. Also could you provide me the master.cfg file (with obfuscated passwords) that is used by the Python buildbot master or tell me if it is in subversion somewhere? I would like to make my script as close as possible to yours, in order to propose a patch for the AIX specific flags that have to be used for compilation on this platform when everything will be stable enough. Regards -- Sébastien Sablé ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue1633863] AIX: configure ignores $CC
Hi Martin,
I have started to correct quite a lot of issues I have with Python on
AIX, and since I had to test quite a lot of patchs, I though it would be
more convenient to setup a buildbot for that platform.
So I now have a buildbot environment with 2 slaves (AIX 5.3 and 6.1)
that builds and tests Python (branch py3k) with both gcc and xlc (the
native AIX compiler) (I have 4 builders (py3k-aix6-xlc,
py3k-aix5-xlc, py3k-aix6-gcc, py3k-aix5-gcc).
I expect to add 4 more builders for branch 2.7 in coming days.
I would like to share the results of this buildbot to the Python
community so that issues with AIX could be addressed more easily.
R. David Murray pointed me to the page on the python wiki concerning
buildbot. It is stated there that is is possible to connect some slaves
to some official Python buildbot master.
Unfortunately, I don't think this solution is possible for me: I don't
think the security team in my company would appreciate that a server
inside our network runs some arbitrary shell commands provided by some
external source. Neither can I expose the buildbot master web interface.
Also I had to customize the buildbot rules in order to work with some
specificities of AIX (see attached master.cfg), and I can't guarantee
that this buildbot will run 24 hours a day; I may have to schedule it
only once at night for example if it consumes too much resources.
(And the results are very unstable at the moment, mostly because of
issue 9862).
On the other hand, I could upload the build results with rsync or scp
somewhere or setup some MailNotifier if that can help.
How do you think I could share those results?
regards
--
Sébastien Sablé
Le 15/09/2010 23:28, R. David Murray a écrit :
R. David Murray added the comment:
Sébastien, you could email Martin (tracker id loewis) about adding your
buildbot to our unstable fleet (or even to stable if it is stable; that is, the
tests normally pass and don't randomly fail). As long as you are around to
help fix bugs it would be great to have an aix buildbot in our buildbot fleet.
(NB: see also http://wiki.python.org/moin/BuildBot, which unfortunately is a
bit out of date...)
--
nosy: +r.david.murray
___
Python trackerrep...@bugs.python.org
http://bugs.python.org/issue1633863
___
# -*- python -*-
# ex: set syntax=python:
# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory (although the filename
# can be changed with the --basedir option to 'mktap buildbot master').
# It has one job: define a dictionary named BuildmasterConfig. This
# dictionary has a variety of keys to control different aspects of the
# buildmaster. They are documented in docs/config.xhtml .
# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}
### BUILDSLAVES
# the 'slaves' list defines the set of allowable buildslaves. Each element is
# a BuildSlave object, which is created with bot-name, bot-password. These
# correspond to values given to the buildslave's mktap invocation.
from buildbot.buildslave import BuildSlave
c['slaves'] = [BuildSlave(phenix, bot1passwd, max_builds=1),
BuildSlave(sirius, bot2passwd, max_builds=1)]
# to limit to two concurrent builds on a slave, use
# c['slaves'] = [BuildSlave(bot1name, bot1passwd, max_builds=2)]
# 'slavePortnum' defines the TCP port to listen on. This must match the value
# configured into the buildslaves (with their --master option)
c['slavePortnum'] = 9989
### CHANGESOURCES
# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes. Any class which implements IChangeSource can be
# put here: there are several in buildbot/changes/*.py to choose from.
from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()
# For example, if you had CVSToys installed on your repository, and your
# CVSROOT/freshcfg file had an entry like this:
#pb = ConfigurationSet([
#(None, None, None, PBService(userpass=('foo', 'bar'), port=4519)),
#])
# then you could use the following buildmaster Change Source to subscribe to
# the FreshCVS daemon and be notified on every commit:
#
#from buildbot.changes.freshcvs import FreshCVSSource
#fc_source = FreshCVSSource(cvs.example.com, 4519, foo, bar)
#c['change_source'] = fc_source
# or, use a PBChangeSource, and then have your repository's commit script run
# 'buildbot sendchange', or use contrib/svn_buildbot.py, or
# contrib/arch_buildbot.py :
#
#from buildbot.changes.pb import PBChangeSource
#c['change_source'] = PBChangeSource()
# If you wat to use SVNPoller, it might look something like
# # Where to get source code changes
# from buildbot.changes.svnpoller import SVNPoller
# source_code_svn_url='https://svn.myproject.org/bluejay/trunk
