Re: [Python-Dev] new security doc using object-capabilities
Hi Brett, On Sat, Jul 22, 2006 at 10:33:19AM -0700, Brett Cannon wrote: Thanks for the link, Armin. Since you guys don't have the import restrictions the CPython version would have and just have different coding needs for RPython obviously I can't just do a blind copy. But I will definitely take a look as I develop. Maybe you guys can even help to lower the duplication if it makes sense for you. Yes, it should be possible to abstract the common logic in some way, using some kind of interface for all OS inspection and 'sys.modules' manipulations. BTW, do you guys happen to have extra tests from import? Yes, there is http://codespeak.net/svn/pypy/dist/pypy/module/__builtin__/test/test_import.py which will also need a bit of rewriting, but that should be straightforward. A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
Hi David, hi Brett, On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote: If I understand correctly, the proposal is that any incompatible changes to the language would apply only in sandboxed interpreters. So there is no reason why support for these couldn't go into the main branch. That's what I originally thought too, but Brett writes: Implementation Details An important point to keep in mind when reading about the implementation details for the security model is that these are general changes and are not special to any type of interpreter, sandboxed or otherwise. That means if a change to a built-in type is suggested and it does not involve a proxy, that change is meant Python-wide for *all* interpreters. So that's why I'm starting to worry that Brett is proposing to change the regular Python language too. However, Brett, you also say somewhere else that backward compatibility is not an issue. So I'm a bit confused actually... Also, I hate to sound self-centered, but I should point out somewhere that PyPy was started by people who no longer wanted to maintain a fork of CPython, and preferred to work on building CPython-like variants automatically. Many of the security features you list would be quite easier to implement and maintain in PyPy than CPython -- also from a security perspective: it is easier to be sure that some protection is complete, and remains complete over time, if it is systematically generated instead of hand-patched in a dozen places. A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Document performance requirements?
Armin Rigo wrote: I think that O-wise the current CPython situation should be documented as a minimal requirement for implementations of the language, with just one exception: the well-documented don't rely on this hack in 2.4 to make repeated 'str += str' amortized linear, for which the 2.3 quadratic behavior is considered compliant enough. I suppose that allowing implementations to provide better algorithmic complexities than required is fine, although I can think of some problems with that (e.g. nice and efficient user code that would perform horribly badly on CPython). I'm not sure big-O tells the whole truth. For instance, do we want to allow an implementation to use a hash table as underlying type for a list? It would match big-O requirements, but would still be slower than a plain array because of higher overhead of implementation (higher constant factor). And if this is allowed, I would like to find in CPython tutorials and documentations a simple statement like: to implement the list and match its requirements, CPython choose a simple array as underlying data structure. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
Hi,
Has anyone else built Python with Visual Studio 2005 and played around
with Profile Guided Optimization? I had to build Python from source w/
VS 2005 as I had a few .pyd's built with VS 2005 that I wanted to load;
I ended up playing around with Profile Guided Optimization, running
``python.exe pystones.py'' to collect call-graph data after
python.exe/Python24.dll had been instrumented, then recompiling with the
optimizations fed back in.
Results were interesting, an average speedup of around 33% was
noticeable:
ActiveState 2.4.3 python.exe:
C:\Python24python.exe Lib\test\pystone.py
Pystone(1.1) time for 5 passes = 0.980119 This machine
benchmarks at
51014.2 pystones/second
The python compiled from branches/release24-maint with VS 2005 + profile
guided optimization:
C:\Python24python.exe Lib\test\pystone.py
Pystone(1.1) time for 5 passes = 0.73261 This machine benchmarks
at
68249.2 pystones/second
Is there any motivation in the Win32 Python dev camp to switch from VC6
to VS 2005?
FWIW, although there were a shed-load of warnings when compiling python
and pythoncore (and a lot more errors when compiling other modules), I
only had to apply one patch to get it working well enough to run
pystone.py. Without this patch, the VC8 CRT aborts at runtime as soon
as an invalid signal is passed to signal(); which is inevitable given
the current code in the initsignal() method:
for (i = 1; i NSIG; i++) {
void (*t)(int);
t = PyOS_getsig(i);
Regards,
Trent.
--
http://www.onresolve.com
Index: signalmodule.c
===
--- signalmodule.c (revision 47196)
+++ signalmodule.c (working copy)
@@ -280,7 +280,21 @@
{NULL, NULL} /* sentinel */
};
+#define WIN32VS2005HACK
+#ifdef WIN32VS2005HACK
+#include stdio.h
+#include stdlib.h
+#include crtdbg.h
+void dummy_handler(const wchar_t *exp,
+ const wchar_t *fn,
+ const wchar_t *file,
+ unsigned int line,
+ uintptr_t reserved)
+{
+}
+#endif
+
PyDoc_STRVAR(module_doc,
This module provides mechanisms to use signal handlers in Python.\n\
\n\
@@ -339,6 +353,12 @@
goto finally;
Py_INCREF(IntHandler);
+#ifdef WIN32VS2005HACK
+(void)_set_invalid_parameter_handler(dummy_handler);
+_CrtSetReportMode(_CRT_ASSERT, 0);
+#endif
+
+
Handlers[0].tripped = 0;
for (i = 1; i NSIG; i++) {
void (*t)(int);
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Socket Timeouts patch 1519025
I request a review of my patch (1519025) to get socket timeouts to work properly with errors and signals. I don't expect this patch would make it into 2.5, but perhaps it could be in 2.5.1, as it fixes a long-standing bug. I know that people are busy with getting 2.5 out the door, but it would be helpful for me to know if my current patch is OK before I start on another patch to make socket timeouts more useful. There is also a version of the patch for 2.4, which would make yum nicer in Fedora 4 and 5, and I think that passing a review would make the patch more acceptable to Fedora's maintainers. My next patch will, if it works, make socket timeouts easier to use per-thread, allow for the timing of entire operations rather than just timing transaction phases, allow for setting an acceptable rate for file transfers, and should be completely backward compatible, in that old code would be unaffected and new code would work as well as possible now on older unpatched versions. That's my plan, anyway. It would build on my current patch, at least in its principles. TonyN.:' mailto:[EMAIL PROTECTED] ' http://www.georgeanelson.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
On 7/23/06, Armin Rigo [EMAIL PROTECTED] wrote: Hi David, hi Brett,On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote: If I understand correctly, the proposal is that any incompatible changes to the language would apply only in sandboxed interpreters. So there is no reason why support for these couldn't go into the main branch.That's what I originally thought too, but Brett writes:Implementation DetailsAn important point to keep in mind when reading about the implementation details for the security model is that these aregeneral changes and are not special to any type of interpreter,sandboxed or otherwise.That means if a change to a built-in type is suggested and it does not involve a proxy, that change is meantPython-wide for *all* interpreters.So that's why I'm starting to worry that Brett is proposing to changethe regular Python language too. Yes, I am proposing changing some constructors and methods on some built-in types for the regular languages. That's it. No new keywords or major semantic changes and such. If I make changes just for sandboxed interpreters it changes the general approach of the security model by then requiring an identity check to see if the interpreter is sandboxed or not. However, Brett, you also say somewhereelse that backward compatibility is not an issue.So I'm a bit confused actually...Since this is my Ph.D. dissertation first and foremost, I am not going to tie my hands in such a way that I have to make too much of a compromise in order for this to work. I obviously don't want to change the feel of Python, but if I have to remove the constructor for code objects to prevent evil bytecode or __subclasses__() from object to prevent poking around stuff, then so be it. For this project, security is trumpeting backwards-compatibility when the latter is impossible in order to have the former. I will obviously try to minimize it, but something that works at such a basic level of the language is just going to require some changes for it to work. Also, I hate to sound self-centered, but I should point out somewherethat PyPy was started by people who no longer wanted to maintain a fork of CPython, and preferred to work on building CPython-like variantsautomatically.Many of the security features you list would be quiteeasier to implement and maintain in PyPy than CPython -- also from asecurity perspective: it is easier to be sure that some protection is complete, and remains complete over time, if it is systematicallygenerated instead of hand-patched in a dozen places.It doesn't sound self-centered. =) Problem is that my knowledge base is obviously all in CPython so my startup costs are much lower than if I tried this in PyPy. Plus there is the point of embedding this into Firefox (possibly) eventually. Does PyPy support embedding yet at the C level? -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
Trent Nelson wrote: Has anyone else built Python with Visual Studio 2005 and played around with Profile Guided Optimization? Yes, there was some work at the recent Need for Speed sprint. Python 2.5 has a PCBuild8 directory (for VS 2005) with a specific project for PGO. Results were interesting, an average speedup of around 33% was noticeable: Yes, they are. Is there any motivation in the Win32 Python dev camp to switch from VC6 to VS 2005? I think Martin decided to keep VC71 (Visual Studio .NET 2003) for another release cycle. Given the impressive results of VC8 with PGO, and the fact that Visual Studio Express 2005 is free forever, I would hope as well for the decision to be reconsidered. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote: I obviously don't want to change the feel of Python, but if I have to remove the constructor for code objects to prevent evil bytecode or __subclasses__() from object to prevent poking around stuff, then so be it. For this project, security is trumpeting backwards-compatibility when the latter is impossible in order to have the former. I will obviously try to minimize it, but something that works at such a basic level of the language is just going to require some changes for it to work. Zope 3's sandboxing machinery manages to handle securing these things without any language changes. So, declaring it impossible to manage without backward compatibility seems inappropriate, or at least incorrect. But perhaps there is something I'm missing? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
Phillip J. Eby wrote: At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote: I obviously don't want to change the feel of Python, but if I have to remove the constructor for code objects to prevent evil bytecode or __subclasses__() from object to prevent poking around stuff, then so be it. For this project, security is [trumping] backwards-compatibility when the latter is impossible in order to have the former. I will obviously try to minimize it, but something that works at such a basic level of the language is just going to require some changes for it to work. Zope 3's sandboxing machinery manages to handle securing these things without any language changes. So, declaring it impossible to manage without backward compatibility seems inappropriate, or at least incorrect. ... if Zope's sandboxing is secure. I haven't done a security review of it, but your argument assumes that it is. In any case, Zope's sandboxing is not capability-based. -- David Hopwood [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
On Jul 23, 2006, at 4:41 PM, Giovanni Bajo wrote: I think Martin decided to keep VC71 (Visual Studio .NET 2003) for another release cycle. Given the impressive results of VC8 with PGO, and the fact that Visual Studio Express 2005 is free forever, I would hope as well for the decision to be reconsidered. Wasn't there a Free Forever 2003 edition too, which has since completely disappeared? Why do you think that MS won't stop distributing the Free Forever VS 2005 once VS 2005+1 comes out, the same way they did the 2003 one? James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
Brett Cannon wrote: On 7/23/06, Armin Rigo [EMAIL PROTECTED] wrote: Hi David, hi Brett, On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote: If I understand correctly, the proposal is that any incompatible changes to the language would apply only in sandboxed interpreters. So there is no reason why support for these couldn't go into the main branch. That's what I originally thought too, but Brett writes: Implementation Details An important point to keep in mind when reading about the implementation details for the security model is that these are general changes and are not special to any type of interpreter, sandboxed or otherwise. That means if a change to a built-in type is suggested and it does not involve a proxy, that change is meant Python-wide for *all* interpreters. So that's why I'm starting to worry that Brett is proposing to change the regular Python language too. Yes, I am proposing changing some constructors and methods on some built-in types for the regular languages. That's it. No new keywords or major semantic changes and such. If I make changes just for sandboxed interpreters it changes the general approach of the security model by then requiring an identity check to see if the interpreter is sandboxed or not. I assume that the extent of incompatible changes would be limited as much as possible. So the only checks would be in operations that are directly affected by whatever incompatible changes are made. The performance and complexity costs of this are likely to be small -- or at least should not be assumed to be large before having hammered out a more detailed design. Suppose, for the sake of argument, that we introduced private methods and attributes. If an attribute in an existing standard library class was changed to be private, then code depending on it would break. But if there were a notion of a compatibility private attribute that acts as private only in a sandboxed interpreter, then no code running in an unprotected interpreter would break. -- David Hopwood [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
Giovanni Bajo [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] that Visual Studio Express 2005 is free forever, I would hope as well for the decision to be reconsidered. But is it freely redistributable forever? Or even now? I have the 2003 toolkit sitting on my disk, but I am apparently not supposed to send it to anyone else. tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
Trent Nelson: I ended up playing around with Profile Guided Optimization, running ``python.exe pystones.py'' to collect call-graph data after python.exe/Python24.dll had been instrumented, then recompiling with the optimizations fed back in. It'd be an idea to build a larger body of Python code to run the profiling pass on so it doesn't just optimize the sort of code in pystone which is not very representative. Could run the test suite as it would have good coverage but would hit exceptional cases too heavily. Other compilers (Intel?) support profile directed optimization so would also benefit from such a body of code. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
James Y Knight [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] On Jul 23, 2006, at 4:41 PM, Giovanni Bajo wrote: I think Martin decided to keep VC71 (Visual Studio .NET 2003) for another release cycle. Given the impressive results of VC8 with PGO, and the fact that Visual Studio Express 2005 is free forever, I would hope as well for the decision to be reconsidered. Wasn't there a Free Forever 2003 edition too, which has since completely disappeared? Why do you think that MS won't stop distributing the Free Forever VS 2005 once VS 2005+1 comes out, the same way they did the 2003 one? I am not aware of any full free ofeering based on 2003. There were ways to get the command line tools, but the GUI's were not available. Most of the freely available command line utilities are still available from microsoft if you find the correct page. I will note that according to the FAQ, Profile Guided Optimizations will not be available with Express. Because the compiler is the same, this means the tools to generate a profile are all that is missing. Express should still be able to build using the optimized profile. Microsoft as a general rule, does not go after people distributing products that Microsoft has labeled free, even after Microsoft no longer distributes that product. So the express editions will continue to be available long into the future if 2005+1 does not have a free version. (The logic behind this is that Microsoft would have a hard time explaining to a jury how somebody can pirate software that is available at no cost.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
Neil Hodgson [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Trent Nelson: I ended up playing around with Profile Guided Optimization, running ``python.exe pystones.py'' to collect call-graph data after python.exe/Python24.dll had been instrumented, then recompiling with the optimizations fed back in. It'd be an idea to build a larger body of Python code to run the profiling pass on so it doesn't just optimize the sort of code in pystone which is not very representative. Could run the test suite as it would have good coverage but would hit exceptional cases too heavily. Other compilers (Intel?) support profile directed optimization so would also benefit from such a body of code. GCC suppost profiling optimized code. One caveat is that the profile-enabled builds ause a GPL'd library. Not a problem for Python right now as it is GPL-compatible, but a caveat non-the-less. That does not apply to the final optimized app. Anyway the flags used are: -fprofile-generate and -fprofile-use. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
Joe Smith wrote: Microsoft as a general rule, does not go after people distributing products that Microsoft has labeled free, even after Microsoft no longer distributes that product. So the express editions will continue to be available long into the future if 2005+1 does not have a free version. Interesting. So people can do the same with the free 2003 version. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
On 7/23/06, Phillip J. Eby [EMAIL PROTECTED] wrote: At 11:07 PM 7/23/2006 +0100, David Hopwood wrote:Phillip J. Eby wrote:[snip] Brett's securing_python.txt don't refer to or cite Zope in any way, butrather relies on broad and unsupported assertions about what can or can'tbe done with Python.I hope he isn't doing the same in his thesis, as this is rather like writing about one's new theory of how to have a worldwideball-kicking contest without making any reference as to how one's theorycompares with the World Cup.The design doc is not meant to be taken as any sort of draft of my thesis. I did read that link you sent me, Philip, but it was hard to follow. So I used Google to find another reference that explained it to me much more clearly. securing_python.txt is meant to explain what I am planning to python-dev so that if someone sees some fatal flaw they can speak up and let me know, not as a thorough comparison of why my approach is better than anyone other one. I'm not saying Zope is better or worse.I'm simply saying that in abusiness context, a failure to compare and contrast a proposed build solution to show how it would be better than a well-established availablebuy solution would be called something like lack of due diligence.Ithink in the academic context it might be called something like failure to cite, but the general idea is the same, i.e., not doing your homework.:)In other words, if the solution being proposed is better than what Zopedoes, the appropriate thing in business is to show the reasons why, and the appropriate thing in science is to state a hypothesis regarding thedifferences, and then perform an experiment to either prove or disprove it.I am not going to write out a blow-by-blow comparison right now. It will come with the thesis. And I am not expecting my approach or code to be checked in blindly anyway. In any case, Zope's sandboxing is not capability-based.You're right: you haven't done a review of it.:)If you had, you'd know that one proxy plus one namechecker equals one capability.In other words,you could take the restricted interpreter, the proxy mechanism, and thenamechecker and leave most of the rest alone, and you'd have your capability system.Then you could focus more time and attention on theparts of the problem that Zope *doesn't* solve, instead of reinventing theones that it already does.Right, but I am trying to remove the need for a namechecker which makes it an object-capabilities system. Now, if Brett believes that changing the Python language is a *better* way to implement capabilities than using proxies to implement them, thengreat.His paper should explain why, and (presumably) include experimentalresults to show that they're either better or worse than Zope's approach based on some criteria.The same information is relevant to Python-Dev asto what is an appropriate approach to support sandboxing in CPython.Whatare the advantages of a built-in approach versus an add-on approach?Are there interpreter facilities that could be added to shore up any awkwardaspects of Zope's approach?(Whatever those might be.)I think people are starting to lose sight of the purpose of the doc I wrote. It was to explain what I was doing for people to see if there was any fatal flaw and to keep people updated on what I am planning on doing. It is not meant to convince anyone that my way is the best way yet. I am not even going to attempt that until I have working code. For example, one part of Zope's approach uses a custom compiler and custom builtins in order to redefine how attribute access works in certaincases.Could these customizations be replaced with options built into thePython compiler and interpreter?What improvements would that result in? Part of my point is to help alleviate the need for custom anything. Simply handwaving all of these questions away, however, with broadassertions of superiority and without even attempting to compare the newwork to Zope's existing work is really not acceptable for academia ORPython development. For the record: I have no personal interest in Zope's security system.Ididn't develop it and haven't had the need to use it, myself.I oncereviewed some of the code and offered some minor suggestions, mainly regarding performance improvement.My only axe to grind in this matter iswhat I've already stated: I think it would be crazy (in the monumentalwaste of resources sense) to consider putting *any* sandboxing system into CPython without tapping the Zope team's experiences.For example: havingimplemented such a system, what compiler or interpreter changes would'vemade the job easier?Meanwhile, what Brett does or doesn't put in his thesis is between him and his advisor, but what gets put into Python shouldn't be based on ignoringthe existing field experience and state of the art.There is no ignoring of anything. I understand their basic approach and I want to try another one. I like the fundemental design difference of object-capabilities and so I am going
Re: [Python-Dev] Python 2.4, VS 2005 Profile Guided Optmization
Joe Smith wrote: Microsoft as a general rule, does not go after people distributing products that Microsoft has labeled free, even after Microsoft no longer distributes that product. But if the licence agreement technically forbids redistribution, it doesn't seem like a good idea to rely on Microsoft turning a blind eye to that. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
