Re: [Python-Dev] building a module catalogue with buildbot

[Martin v. Löwis]

Andrew Bennetts wrote:
> A limited solution is just to make the script put the files where they will be
> published by something other than buildbot.  e.g. Twisted's docs are built 
> from
> SVN by one of our buildslaves, and placed in a directory published at
> http://twistedmatrix.com/users/warner/doc-latest/

Right. Whether this could work for us depends on how the files could be
transferred. I would not want to see rsync, ftp, scp or some such to
put files on python.org, but I could imagine getting them "through"
the twisted connection, if that were possible somehow (i.e. with the
master itself putting the files into the places where I want them).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] building a module catalogue with buildbot

[Andrew Bennetts]

On Thu, Jan 12, 2006 at 07:19:08AM +0100, "Martin v. Löwis" wrote:
> Fredrik Lundh wrote:
> > My initial thought was that we could ask alpha testers to run this script on
> > their alpha builds, and report back, but it just struck me that the 
> > "buildbot"
> > already builds stuff on a couple of interesting platforms.
> > 
> > Can buildbot deal with custom test/validation scripts, and collect the 
> > output
> > somewhere ?
> 
> Collecting the output is the real challenge. Basically, you are
> restricted to putting stdout of a process into a file. So a result
> that is a set of linked HTML files is not supported.

A limited solution is just to make the script put the files where they will be
published by something other than buildbot.  e.g. Twisted's docs are built from
SVN by one of our buildslaves, and placed in a directory published at
http://twistedmatrix.com/users/warner/doc-latest/

-Andrew.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] building a module catalogue with buildbot

[Martin v. Löwis]

Fredrik Lundh wrote:
> My initial thought was that we could ask alpha testers to run this script on
> their alpha builds, and report back, but it just struck me that the "buildbot"
> already builds stuff on a couple of interesting platforms.
> 
> Can buildbot deal with custom test/validation scripts, and collect the output
> somewhere ?

Collecting the output is the real challenge. Basically, you are
restricted to putting stdout of a process into a file. So a result
that is a set of linked HTML files is not supported.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Buildbot-devel] Re: buildbot

[Jean-Paul Calderone]

On Tue, 10 Jan 2006 09:18:59 -0800, Stephen Davis <[EMAIL PROTECTED]> wrote:
>> The reason I want static pages is for security concerns. It is not
>> easy whether buildbot can be trusted to have no security flaws,
>> which might allow people to start new processes on the master,
>> or (perhaps worse) on any of the slaves.
>
>I have security concerns as well, but not in buildbot itself.  My
>project is restricted even withinz the company I work for so I need
>the buildbot web server to only be available to certain people.
>HTTPS access would be nice too.  TwistedWeb doesn't seem to have
>support for either HTTPS or authentication so I've been forced to
>"hide" it by putting it on a non-standard port.  Very weak.

Inexcusably weak.  Fortunately not exactly accurate, though: Twisted's web 
server supports both HTTPS and authentication.  I won't bore the list with the 
details, but you can read more here: 

 and 


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] r42015 - peps/trunk

[David Goodger]

> Author: david.goodger
> Date: Thu Jan 12 04:33:16 2006
> New Revision: 42015
> 
> Modified:
>peps/trunk/   (props changed)
> Log:
> add external link to Docutils public repo -- always up-to-date

I just deleted the static copy of the "docutils" directory from the "peps"
repository, and added in an external link (svn:externals 'docutils
svn://svn.berlios.de/docutils/trunk/docutils/docutils').  This way, the external
code should always be up-to-date.  You may need to manually delete your
peps/trunk/docutils directory to get this to work though -- SVN leaves
subdirectories behind which hinder the externals update.

Please let me know if this causes any problems.  Thanks.

-- 
David Goodger 



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] building a module catalogue with buildbot

[Brian Warner]

> PS If there was a method on ShellCommand, adding a step would be simpler,
> e.g.,
>
>   def addStep(self, cls, **kwds):
> self.steps.append((cls, kwds))

Ooh! I like that.

> then we could do: f.addStep(Catalog, command=Catalog.command)

It would be even simpler: f.addStep(Catalog). The command= argument is
provided by the class-level attributes of your Catalog class (it can be set
by either an attribute or by a constructor argument, but if it's always going
to be the same thing then the attribute is easier).

If you make a patch for this, I'll commit it. If not, I'll write one before I
get the next release out.

thanks,
 -Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PATCH] Fix dictionary subclass semantics when used as global dictionaries

[Crutcher Dunnavant]

Is there any objection to this patch? Any support?

On 1/10/06, Crutcher Dunnavant <[EMAIL PROTECTED]> wrote:
> 1402289
>
> On 1/10/06, Aahz <[EMAIL PROTECTED]> wrote:
> > On Tue, Jan 10, 2006, Crutcher Dunnavant wrote:
> > >
> > > There is an inconsistancy in the way that dictionary subclasses behave
> > > when they are used as as namespaces in execs.
> > >
> > > Basically, while python 2.4 permits the usage of dictionary subclasses
> > > for local environments, it still bypasses the subclass functions and
> > > uses the C API for global environments. The attached patch (and
> > > unittest!) addresses this issue.
> >
> > Please submit the patch to SourceForge and report back with the SF ID.
> > --
> > Aahz ([EMAIL PROTECTED])   <*> http://www.pythoncraft.com/
> >
> > "19. A language that doesn't affect the way you think about programming,
> > is not worth knowing."  --Alan Perlis
> >
>
>
> --
> Crutcher Dunnavant <[EMAIL PROTECTED]>
> monket.samedi-studios.com
>


--
Crutcher Dunnavant <[EMAIL PROTECTED]>
monket.samedi-studios.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ConfigParser to save with order

[Facundo Batista]

2006/1/11, Tony Meyer <[EMAIL PROTECTED]>:

> Remember that there has been a lot of discussion about how
> ConfigParser should work in the past; for example (ignoring c.l.p):
>
> 
> 
> 

My god. Read it all (which took me near a thermo flask (or whatever
you call to an insulated bottle to keep liquids cold or hot) of
"mate"!).

I see two paths here:

- Good one
- Bad one

No, sorry, seriously:

- Rewrite ConfigParser entirely.
- Apply my patch.

The first option is the better, though it's the one which will take
more work. Someone needs to make a PEP. Michael Chermside, in the
first mail of the first link that you sent, was working in a solution,
but don't know what is done. In this case, both patchs I talked before
should be rejected.

The second one is the quickest.

Both paths solves my problem. I think we should go with the second
option in the short term, and with the first one in the long term.

What do you think?

.Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Delaney, Timothy (Tim)]

"Martin v. Löwis" wrote:

> So as for dealing with it "somehow": I would make ctypes a dynamically
> loaded module (ctypes.pyd), so administrators could remove it, and
> I could also make it a separate option in the Windows installer,
> so administrators could reject to install it.

I like this solution. Of course, Thomas (author of both py2exe and ctypes) may 
like the ability to have ctypes built into python.dll ...

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] building a module catalogue with buildbot

[Trent Mick]

> > Does that make sense?  We would just need /f's script in SVN.
> 
> in python/Tools/something or sandbox/something ?

python/Doc/tools/something?

Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot: doing occasional full builds

[Brian Warner]

> nice-ing the slave process at startup would be the way to do it, right?

Yup. We run the twisted buildslaves under 'nice', and it works pretty well.
It also reveals problems in tests that use absolute timeouts which fail when
the test runs slower than the author thought it was supposed to.

cheers,
 -Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Martin v. Löwis]

Guido van Rossum wrote:
> On the other hand it breaks one of the most fundamental Python
> guidelines: if you get a core dump (segfault etc.) it's a bug in
> Python or in a 3rd party extension, not in *your* Python code. An
> exception would have to be made for any code that uses ctypes, as it
> is usually trivial to cause core dumps with ctypes (I'd venture it's
> hard to avoid them ;-).
> 
> I don't expect this to count against including ctypes; but I do want
> it to be dealt with somehow!

I think the strongest point *for* ctypes is the inclusion of dl
in the source distribution, which has the very same flaws as
ctypes in terms of crashability.

So as for dealing with it "somehow": I would make ctypes a dynamically
loaded module (ctypes.pyd), so administrators could remove it, and
I could also make it a separate option in the Windows installer,
so administrators could reject to install it.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New PEP: Using ssize_t as the index type

[Martin v. Löwis]

Tim Peters wrote:
> This reminded me that I still owe you a reply about s# and t# format
> codes.  It occurred to me that I've never used them, and probably
> never will, so I really don't care how they work:  I'm only really
> worried about widespread ugliness, meaning wide enough that it touches
> me ;-)

Ok, given that Armin also "gave in", I think I keep that at a macro.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot: doing occasional full builds

[Martin v. Löwis]

Trent Mick wrote:
> Specifically are you concerned about the readability of the waterfall
> page or other things (maintainability or something)? If the former,
> perhaps we could get some mileage out of the query args that Brian
> mentioned:

Indeed - that would be a solution. However, I wonder how many people
would look at the (say) nightly builds.

Anyway, I'll try that.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot: doing occasional full builds

[Martin v. Löwis]

[EMAIL PROTECTED] wrote:
> One concern might be performance.  All buildbot slaves are contributed
> hardware.  I don't mind the load on my Mac (it's a dual processor g5), but
> it may be an issue for other people.  Does/can buildbot run at a lower
> priority or is it just a matter of nice-ing the slave process at startup?

nice-ing the slave process at startup would be the way to do it, right?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] building a module catalogue with buildbot

[Fredrik Lundh]

Neal Norwitz wrote:

> Does that make sense?  We would just need /f's script in SVN.

in python/Tools/something or sandbox/something ?





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot: doing occasional full builds

[Trent Mick]

[EMAIL PROTECTED] wrote]
> 
> Trent> Specifically are you concerned about the readability of the
> Trent> waterfall page or other things (maintainability or something)? If
> Trent> the former, perhaps we could get some mileage out of the query
> Trent> args that Brian mentioned:
> 
> One concern might be performance.  All buildbot slaves are contributed
> hardware.  I don't mind the load on my Mac (it's a dual processor g5), but
> it may be an issue for other people.  Does/can buildbot run at a lower
> priority or is it just a matter of nice-ing the slave process at startup?

As (I think) Brian mentioned, the scheduler for the (proposed) "full
builder" could be setup to just build, say, once per day or only a couple
of times per day where as the "incremental builder" would trigger for
every SVN checkin.

Nicing could probably also be done.

Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot: doing occasional full builds

[Neal Norwitz]

On 1/11/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> One concern might be performance.  All buildbot slaves are contributed
> hardware.  I don't mind the load on my Mac (it's a dual processor g5), but
> it may be an issue for other people.

I've "contributed" 2 machines.  One is my personal box, the other is
the PSFs.  The PSF box is very lightly used except by automated stuff,
so it's not a problem.  My box is a dual opteron and it really isn't a
problem for me right now.  I suppose if it was building all the time,
I might mind, but so far, there's no issues.

Hopefully over time we will get more people contributing other boxes
and this might become a bigger issue.  But right now I don't think
it's a problem.

n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot: doing occasional full builds

[skip]

Trent> Specifically are you concerned about the readability of the
Trent> waterfall page or other things (maintainability or something)? If
Trent> the former, perhaps we could get some mileage out of the query
Trent> args that Brian mentioned:

One concern might be performance.  All buildbot slaves are contributed
hardware.  I don't mind the load on my Mac (it's a dual processor g5), but
it may be an issue for other people.  Does/can buildbot run at a lower
priority or is it just a matter of nice-ing the slave process at startup?

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot: doing occasional full builds

[Trent Mick]

[Martin v. Loewis wrote]
> Brian Warner wrote:
> > That would work, although to be honest the buildbot is more about repeatable
> > builds. My first suggestion would be to have two separate Builders, one of
> > which does incremental builds, the other which does full (from-scratch)
> > builds.
> 
> My concern is that then the number of builders would go up from 8 to 16,
> even though it is just four different machines that we use.

Specifically are you concerned about the readability of the waterfall
page or other things (maintainability or something)? If the former,
perhaps we could get some mileage out of the query args that Brian
mentioned:

[Brian Warner wrote]
> I should mention a feature that you might find useful: you can add
> some query arguments to the top-level Waterfall page's URL to
> restrict the set of Builders that are displayed. If you append
> "?show=full-2.3", you'll wind up with a waterfall display that
> only contains the "full-2.3" Builder's column.  Appending
> "?show=full-2.3&show=full-2.4" will show two columns, etc. You can
> also create multiple Waterfall instances, each with a different
> set of "categories", and use that to manage the complexity of
> displaying status for lots of Builders. (with the current release
> these multiple Waterfalls must all listen on different TCP ports,
> unfortunately, but you can always hide this with the
> reverse-proxy's URL mapping).


Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot: doing occasional full builds

[Martin v. Löwis]

Brian Warner wrote:
> That would work, although to be honest the buildbot is more about repeatable
> builds. My first suggestion would be to have two separate Builders, one of
> which does incremental builds, the other which does full (from-scratch)
> builds.

My concern is that then the number of builders would go up from 8 to 16,
even though it is just four different machines that we use.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] building a module catalogue with buildbot

[Trent Mick]

[Neal Norwitz wrote]
> > [Fredrik Lundh wrote]
> > > Can buildbot deal with custom test/validation scripts, and collect the 
> > > output
> > > somewhere ?
> >
> ...
> It looks like we could define a class similar to Test, such as:
> 
> class Catalog(ShellCommand):
> name = "catalog"
> warnOnFailure = 0 # this was 1, not sure which we want
> description = ["catalog"]
> descriptionDone = ["catalog"]
> command = ["/f cmd here", "and", "other", "flags"]
> 
> The factory is created like this:
> f = factory.GNUAutoconf(source, test=["make","testall"])
> 
> ISTM, we can just add this line:
>f.steps.append(s(Catalog, command=Catalog.command))
> 
> Does that make sense?  We would just need /f's script in SVN.

Yup. Sounds right to me.



> PS  If there was a method on ShellCommand, adding a step would be simpler, 
> e.g.,
> 
>   def addStep(self, cls, **kwds):
> self.steps.append((cls, kwds))
> 
> then we could do: f.addStep(Catalog, command=Catalog.command)
> which would simplify a bunch of code in process/factory.py since it is
> called this way 10 times.  Didn't look elsewhere.

Add a buildbot patch/bug?
http://sourceforge.net/tracker/?func=add&group_id=73177&atid=537003



Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Checking in a broken test was: Re: [Python-checkins]r41940 - python/trunk/Lib/test/test_compiler.py

[Trent Mick]

[Stephen J. Turnbull wrote]
> > "Fredrik" == Fredrik Lundh <[EMAIL PROTECTED]> writes:
> 
> Fredrik> many test frameworks support "expected failures" for this
> Fredrik> purpose.  how hard would it be to add a
> 
> Fredrik> unittest.FailingTestCase
> 
> Fredrik> class that runs a TestCase, catches any errors in it, and
> Fredrik> signals an error ("test foo passed unexpectedly") if it
> Fredrik> runs cleanly ?
> 
> One can do even better than that.  unittest.FailingTestCase should
> (except possibly for platform dependencies) know _how_ the TestCase is
> expected to fail.  You also want to know if the error changes.

How about this:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307970
a better assertRaises() for unittest.py 

When writing unit tests for Python using the standard unittest.py
system the assertRaises() (aka failUnlessRaises()) method is used to
test that a particular call raises the given exception. This recipe
if for assertRaisesEx() that adds three things: (1) the ability to
assert the raised exception's args; (2) the ability to test that the
stringified exception matches a given regular expression; and (3)
much better failure messages.

I haven't read this thread, so apologies is this doesn't really apply to
the discussion.

Cheers,
Trent

-- 
Trent Mick
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Buildbot-devel] Re: buildbot

[Stephen Davis]

> The reason I want static pages is for security concerns. It is not
> easy whether buildbot can be trusted to have no security flaws,
> which might allow people to start new processes on the master,
> or (perhaps worse) on any of the slaves.

I have security concerns as well, but not in buildbot itself.  My  
project is restricted even withinz the company I work for so I need  
the buildbot web server to only be available to certain people.   
HTTPS access would be nice too.  TwistedWeb doesn't seem to have  
support for either HTTPS or authentication so I've been forced to  
"hide" it by putting it on a non-standard port.  Very weak.

I am no networking expert so the suggestions for using a reverse  
proxy are very welcome and I will look into that right away.  Just  
wanted to add my voice to the security concerns.

stephen
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Checking in a broken test was: Re: [Python-checkins]r41940 - python/trunk/Lib/test/test_compiler.py

[Stephen J. Turnbull]

> "Fredrik" == Fredrik Lundh <[EMAIL PROTECTED]> writes:

Fredrik> many test frameworks support "expected failures" for this
Fredrik> purpose.  how hard would it be to add a

Fredrik> unittest.FailingTestCase

Fredrik> class that runs a TestCase, catches any errors in it, and
Fredrik> signals an error ("test foo passed unexpectedly") if it
Fredrik> runs cleanly ?

One can do even better than that.  unittest.FailingTestCase should
(except possibly for platform dependencies) know _how_ the TestCase is
expected to fail.  You also want to know if the error changes.


-- 
Graduate School of Systems and Information Engineering   University of Tsukuba
http://turnbull.sk.tsukuba.ac.jp/Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Economics of Information Communication and Computation Systems
  Experimental Economics, Microeconomic Theory, Game Theory
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Nick Craig-Wood]

On Wed, Jan 11, 2006 at 07:59:03AM +0100, Thomas Heller wrote:
> Another possibility would be to emit a warning when the module (dl or
> ctypes, if included) is imported.
> 
> warnings.warn("Incorrect usage of this module may crash Python",
>   RuntimeWarning, stacklevel=2)

Arrgggh!  No!!  Warnings are one of the things I really hate about
perl and I'd hate to see them infecting python any more than they
already have.

I want my programming language to tell me, the programmer, about stuff
not go blurting to the user behind my back.

Death to warnings ;-)

(Yes I know about the warnings module!)
-- 
Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Ronald Oussoren]

On 11-jan-2006, at 7:59, Thomas Heller wrote:
>>
>>> I'm of the opinion that having a big red
>>> warning at the top of the module documentation that this is a
>>> contributed module, and incorrect use could cause segmentation
>>> faults/crashes, etc would be sufficient.
>>
>> Works for me.
>
> Another possibility would be to emit a warning when the module (dl or
> ctypes, if included) is imported.
>
> warnings.warn("Incorrect usage of this module may crash Python",
>   RuntimeWarning, stacklevel=2)

Yuck. Just adding a warning to the documentation should be good  
enough. Other modules allow you to make your machine unuseable (such  
a fork bomb using os.fork) and those don't warn either.

Ronald

>
> Thomas
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ 
> ronaldoussoren%40mac.com

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Limiting the recursion limit

[Guido van Rossum]

On 1/11/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> >>> sys.setrecursionlimit(1<<30)
> >>> f = lambda f:f(f)
> >>> f(f)
> Segmentation fault
>
> Is there some way that Python can determine that 1<<30 is an unreasonable
> recursion limit?

Yes, but that doesn't help -- there's some value in the middle where
you may or may not get a crash depending on random other stuff that is
going on, and the only way to know is to try. It's easy enough to code
a loop that tries higher and higher values until it finds a crash.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Limiting the recursion limit

[skip]

>>> sys.setrecursionlimit(1<<30)
>>> f = lambda f:f(f)
>>> f(f)
Segmentation fault

Is there some way that Python can determine that 1<<30 is an unreasonable
recursion limit?

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Thomas Wouters]

On Wed, Jan 11, 2006 at 02:54:40PM +0100, Thomas Wouters wrote:

> The pickle vulnerability came up last year, when someone on #python was
> subclassing a builtin type (string or dict, I think the latter) that was
> using a magical invocation of (IIRC) __new__ on unpickle. The subclassed
> __new__ didn't handle this right, so the baseclass __new__ wasn't getting
> called right, and the new object's addressspace was not initialized. This
> lead to crashes. I don't remember the details exactly, and my continuous
> advice of not subclassing builtin types unless you know what you're doing
> solved the issue (there was no actual need to subclass, there), and I have
> no idea whether that specific issue was solved or not, but I'm trying to
> find it again :)

Ah, found it: it was one of the datetime types. It has guards in place (some
back then, python2.3, more since 2.4) so I wasn't able to figure out why it
actually crashed Python, rather than produce a weird date. I couldn't find
anything obviously wrong with the data's handling (although the extra guards
are good.) I'll see if I can reproduce it anyway.

-- 
Thomas Wouters <[EMAIL PROTECTED]>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Michael Hudson]

Guido van Rossum <[EMAIL PROTECTED]> writes:

> On 1/10/06, Thomas Wouters <[EMAIL PROTECTED]> wrote:
>> Sorry, I missed the point I was aiming at, I guess. I wasn't aiming for
>> fixable bugs; I see these things as, with great effort, holding up your foot
>> at an awkward angle so that it ends up right at the business end of your
>> primed and lit medieval cannon. We can jump through hoops to stop those
>> particular examples, but I'd rather spend time and effort to fix *real* bugs
>> instead.
>
> I'm not sure what makes those particular examples less than real bugs.

I think the sys.setrecursionlimit(1<<30) one is fairly well into the
terriotory of "don't do that", isn't it?

Cheers,
mwh

-- 
   today's lesson
   don't strace X in an xterm
-- from Twisted.Quotes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Barry Warsaw]

On Wed, 2006-01-11 at 14:54 +0100, Thomas Wouters wrote:
> On Wed, Jan 11, 2006 at 07:59:50AM -0500, Barry Warsaw wrote:
> 
> > BTW, although I'm pretty sure the answer is "no" (at least, I hope it
> > is), is anyone aware of a situation where the mere importation of a
> > module can cause Python to crash?
> 
> Well, I assume you aren't importing any 'hostile' code, nor running in an
> uncontrolled environment so I guess you mean other than running out of
> memory, or the module you are importing actually executing one of the ways
> to crash Python? 

Correct.

> Or the module being an extension module that crashes on
> import? Or another extension module having corrupted the Python environment
> to a point where a simple import crashes Python? Or a non-extension module
> using one of the vulnerabilities (in, say, marshal, or pickle) to corrupt
> the Python environment? Or stuff in os.environ, like LD_* variables, that
> interfere with library linking? Ponder, ponder, no, can't think of any. :)

Let's keep it to modules in the standard library, although that includes
extension modules, and let's further say that it's a controlled enough
environment that you won't have stray evil modules floating around in
your sys.path.

> The pickle vulnerability came up last year, when someone on #python was
> subclassing a builtin type (string or dict, I think the latter) that was
> using a magical invocation of (IIRC) __new__ on unpickle. The subclassed
> __new__ didn't handle this right, so the baseclass __new__ wasn't getting
> called right, and the new object's addressspace was not initialized. This
> lead to crashes. I don't remember the details exactly, and my continuous
> advice of not subclassing builtin types unless you know what you're doing
> solved the issue (there was no actual need to subclass, there), and I have
> no idea whether that specific issue was solved or not, but I'm trying to
> find it again :)

-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Thomas Wouters]

On Wed, Jan 11, 2006 at 07:59:50AM -0500, Barry Warsaw wrote:

> BTW, although I'm pretty sure the answer is "no" (at least, I hope it
> is), is anyone aware of a situation where the mere importation of a
> module can cause Python to crash?

Well, I assume you aren't importing any 'hostile' code, nor running in an
uncontrolled environment so I guess you mean other than running out of
memory, or the module you are importing actually executing one of the ways
to crash Python? Or the module being an extension module that crashes on
import? Or another extension module having corrupted the Python environment
to a point where a simple import crashes Python? Or a non-extension module
using one of the vulnerabilities (in, say, marshal, or pickle) to corrupt
the Python environment? Or stuff in os.environ, like LD_* variables, that
interfere with library linking? Ponder, ponder, no, can't think of any. :)

The pickle vulnerability came up last year, when someone on #python was
subclassing a builtin type (string or dict, I think the latter) that was
using a magical invocation of (IIRC) __new__ on unpickle. The subclassed
__new__ didn't handle this right, so the baseclass __new__ wasn't getting
called right, and the new object's addressspace was not initialized. This
lead to crashes. I don't remember the details exactly, and my continuous
advice of not subclassing builtin types unless you know what you're doing
solved the issue (there was no actual need to subclass, there), and I have
no idea whether that specific issue was solved or not, but I'm trying to
find it again :)

-- 
Thomas Wouters <[EMAIL PROTECTED]>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Include ctypes into core Python?

[Barry Warsaw]

On Wed, 2006-01-11 at 07:59 +0100, Thomas Heller wrote:

> Another possibility would be to emit a warning when the module (dl or
> ctypes, if included) is imported.
> 
> warnings.warn("Incorrect usage of this module may crash Python",
>   RuntimeWarning, stacklevel=2)

BTW, although I'm pretty sure the answer is "no" (at least, I hope it
is), is anyone aware of a situation where the mere importation of a
module can cause Python to crash?

-Barry



signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] test_curses

[Michael Hudson]

Georg Brandl <[EMAIL PROTECTED]> writes:

>> Well, this still has the faint whiff of impossibility about it.  Are
>> you sure it's setupterm() that's doing the damage?  Can you reproduce
>> interactively?
>
> Yep.
> Alone, the setupterm call [curses.setupterm(sys.__stdout__.fileno())] does
> nothing remarkable, but when it is done inside of curses.initscr() / 
> curses.endwin()
> the terminal is not restored properly.

Ah!  That does indeed make some small amount of sense.

> If setupterm() is to be tested, I think it should be executed before 
> initscr().

Sounds good.  What are you waiting for? :)

Cheers,
mwh

-- 
   Jokes around here tend to get followed by implementations.
-- from Twisted.Quotes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbot

[Brian Warner]

> The reason I want static pages is for security concerns. It is not
> easy whether buildbot can be trusted to have no security flaws,
> which might allow people to start new processes on the master,
> or (perhaps worse) on any of the slaves.

These are excellent points. While it would take a complete security audit to
satisfy the kind of paranoia *I* tend to carry around, I can mention a couple
of points about the buildbot's security design that might help you make some
useful decisions about it:

 The buildmaster "owns" (spelled "pwns" or "0wnz0red" these days, according
 to my "leet-speak phrasebook" :) the buildslaves. It can make them run
 whatever shell commands it likes, therefore it has full control of the
 buildslave accounts. It is appropriate to give the buildslaves their own
 account, with limited privileges.

 The codebase "owns" the buildslaves too: most build commands will wind up
 running './configure' or 'make' or something which executes commands that
 are provided by the checked out source tree.

 Nobody is supposed to "own" the buildmaster: it performs build scheduling
 and status reporting according to its own design and configuration file. A
 compromised codebase cannot make the buildmaster do anything unusual, nor
 can a compromised buildslave. The worst that a buildslave can do is cause a
 DoS attack by sending overly large command-status messages, which can
 prevent the buildmaster from doing anything useful (in the worst case
 causing it to run out of memory), but cannot make it do anything it isn't
 supposed to.

 The top-level IRC functions can be audited by inspecting the command_
 methods, as you've already seen.

 The HTTP status page can be audited similarly, once you know how Twisted-Web
 works (there is a hierarchy of Resource objects, each component of the URL
 path uses Resource.getChild to obtain the child node in this tree, once the
 final child is retrieved then the 'render' method is called to produce
 HTML). The Waterfall resource and all its children get their capabilities
 from two objects: Status (which provides read-only status information about
 all builds) and Control (which is the piece that allows things like "force
 build"). The knob that disables the "Force Build" button does so by creating
 the Waterfall instance with control=None. If you can verify that the code
 doesn't go out of its way to acquire a Control reference through some
 private-use-only attribute, then you can be reasonably confident that it
 isn't possible to make the web server do anything to trigger a build. It's
 not restricted-execution mode or anything, but it's designed with
 capability-based security in mind, and that may help someone who wishes to
 audit it.

 The PBListener status interface is similar: PB guarantees that only remote_*
 methods can be invoked by a remote client, and the PBListener object only
 has a reference to the top-level Status object.

 The slave->master connection (via the 'slaveport') uses PB, so it can be
 audited the same way. Only the remote_* (and perspective_*) methods of
 objects which are provided to the buildslave can be invoked. The buildslaves
 are allowed to do two things to the top-level buildmaster: force a build
 that is run on their own machine, and invoke an empty 'keepalive' method.
 During a build, they can send remote_update and remote_complete messages to
 the current BuildStep: this is how they deliver status information
 (specifically the output of shell commands). By inspecting
 buildbot.process.step.RemoteCommand.remote_update, you can verify that the
 update is appended to a logfile and nothing else.

 PB's serialization is designed specifically to be safe (in explicit contrast
 to pickle). Arbitrary classes cannot be sent over the wire. The worst-case
 attack is DoS, specifically memory exhaustion.

Any application which can talk to the outside world is a security concern.
The tools that we have to insure that these applications only do what we
intended them to do are not as fully developed as we would like (I hang out
with the developers of E, and would love to implement the buildbot in a
capability-secure form of Python, but such a beast is not available right
now, and I'm spending too much time writing Buildbot code to get around to
writing a more secureable language too). So we write our programs in as clear
a way as possible, and take advantage of tools that have been developed or
inspected by people we respect.

These days my paranoia tells me to trust a webserver written in Python more
than one written in C. Buffer overruns are the obvious thing, but another
important distinction is how Twisted's web server architecture treats the URL
as a path of edges in a tree of Resource instances rather than as a pathname
to a file on the disk. I don't need to worry about what kind of URLs might
give access to the master.cfg file (which could contain debugging passwords
or something), as long as I can tell that none of the Resource insta

Re: [Python-Dev] Buildbot questions

[Brian Warner]

> > Ah, but that would require changes to the slaves, right? I would
> > prefer a solution that avoids that.
> 
> I don't think so. In my little test setup I didn't have to make any
> change to the slave.

The "update" and "clobber" mode parameters are implemented on the slave side.
Trent's patch changes which mode setting is used on the master side before
anything is sent to the slaves, so the slaves don't have to know anything
about it.

In general there is as little slave-side code as possible, to reduce the need
to upgrade or change the code on that side. The VC checkout/update operations
are an exception, since they may do several operations in a row (which would
mean a lot more latency to do them without support from code on the slave
side), and because I'm lazy and prefer to implement the algorithms in Python
instead of in Bourne shell :).

 -Brian

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Buildbot: doing occasional full builds

[Brian Warner]

> > > To wipe out the build occassionally you could (presumably) add a
> > > starting step to the Python 'builder' (in the build master.cfg) to 
> > > rm -rf $builddir
> > > every, say, Sunday night.

That would work, although to be honest the buildbot is more about repeatable
builds. My first suggestion would be to have two separate Builders, one of
which does incremental builds, the other which does full (from-scratch)
builds. Both modes are useful to test: I've seen build problems that affected
one but not the other (deleted source files that leave .o files in place,
Makefile changes that don't notice modified files correctly, etc).

If you want to minimize the compile load placed on the build slaves, you
could use the Schedulers to reduce the frequency of triggering the "full"
builder. A subclass of the default Scheduler class could enforce a
minimum-time-between-builds timer, and you could crank it up to only allow
one full build per hour, or per day, or whatever.

The reason I'd encourage you to let every build of a given Builder behave the
same way is that I hope to add more problem-tracking code to the Buildbot,
and that code will work better if builds really are reproducible and stable.
If the tests pass in build #9 and fail in build #10, you should be able to
blame the failure on the changes that went into build #10. If instead the
failure is due to the fact that build #10 just happened to be the one where
you do a clean build instead of an incremental one, then it will be harder to
automatically isolate the cause of the failure.

It isn't a big issue, though. Just thought I'd share what my design
philosophy is so you can decide which direction you want to go. The code is
flexible enough to handle it.

> The current problem with this is that I don't know if the build step
> objects have access to a local data store -- so it could, say, count how
> many builds have been done to know to clobber every tenth one. The
> current code just chooses to clobber for any build on Sunday. Not a very
> satisfying solution.
> 
> Brian,
> Is there a way for build steps to maintain some state?

In general, no.. the long-term status is kept in a separate BuildStatus
object, the idea is that BuildStatus gets persisted but Builds do not. You
can always have your Build subclass use the regular status interface to look
up an earlier BuildStatus, and base some decisions upon the data you find
there, although I haven't really provided a way to set arbitrary attributes
on the BuildStatus.

For this case, however, you can have the Build reach into its BuildStatus
object and look at the .number attribute, and then do a simple %10==0 test.
Not especially elegant but it would get the job done. From a BuildStep you
would use 'self.build.build_status.number % 10 == 0'.

Apart from that, your SVNEx subclass seems feasible. I'd be inclined to have
the subclass always implement this update_and_clobber_occassionally behavior
(rather than providing a special mode= switch for it), but that's just a
usage detail.

cheers,
 -Brian
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com