Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements
On Mar 23, 2014, at 7:03 PM, Barry Warsaw ba...@python.org wrote: On Mar 23, 2014, at 08:00 AM, Skip Montanaro wrote: I'm unclear how this would be better than just biting the bullet and making a 2.8 release. On the one hand, the 2.7.x number suggests (based on the existing release protocol) that it should be a drop-in replacement for earlier 2.7 micro releases. On the other hand, calling it something like ServerPython implies that it's not necessary for network client applications, when, if I read the PEP correctly, it most certainly would be. It seems to me that the problem the PEP addresses can largely be confined to the Python 2.7 standard library and the fact that it's bundled with the language. I want to stick to our no-Python-2.8 pledge, but I wonder if there isn't a middle ground where we fork the stdlib into a separate branch, and apply security specific changes there. Python 2.7.x will always be the standard stdlib. We would never release a specific Python 2.7 + security stdlib release, but downstream developers would be able to overlay this forked stdlib on top of the standard one. Volunteers or corporate sponsors could distribute binary installers with this combination of pure Python 2.7 language + security enhanced stdlib, and Linux distros could do the necessary building and distributing for their own platforms if they so desired. The trick is what do you call this new combination, how do you invoke it, and how do you keep it distinct and independent of the system's standard Python 2.7? Maybe this is some scent of Python 2.8 by another name, but let's never call it Python 2.8. It seems like this and the fork idea are both jumping through hoops to avoid the inevitable - a 2.7 security release or a 2.8 security only release. Especially as I know nick isn't the only one looking to hire FTEs for this specific work as it's actively hurting service providers/distributions and others. If we call it a security only release, and scope/review all patches to that it seems that it's pretty clear what the release is for. 2.x is going to be in the wild for at least another decade as these large legacy codebases are functioning, profitable and dedicating teams of engineers to port them to 3.x doesn't make sense to the business. As new services and things come online we'll see the gradual movement to 3 accelerate especially as people realize clients and service talk better on an interface such as http rather than large monolithic tragedies. -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 marketing document?
fwiw, I'm offering the keys/account/etc for getpython3.com to whomever has the time to keep it fresh and up to date. On Fri, Jan 24, 2014 at 8:36 AM, Wes Turner wes.tur...@gmail.com wrote: Hardly marketing documents, but potentially useful nonetheless: http://docs.python.org/3.4/whatsnew/index.html https://bitbucket.org/gutworth/six/src/tip/six.py https://github.com/nandoflorestan/nine/blob/master/nine/__init__.py http://docs.python.org/3/library/2to3.html https://pypi.python.org/pypi/backports.ssl_match_hostname http://docs.python.org/3.4/library/asyncio.html The new pathlib is pretty cool: http://docs.python.org/3.4/whatsnew/3.4.html#new-modules http://docs.python.org/3.4/library/pathlib.html#module-pathlib Wes Turner On Jan 23, 2014 9:49 PM, Dan Stromberg drsali...@gmail.com wrote: Has anyone published a web page or wiki page about what's great about Python 3.x? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 marketing document?
I'm giving AMK the keys to the kingdom right now: AMK: Feel free to go nuts. Email me your public key On Fri, Jan 24, 2014 at 12:01 PM, Mark Lawrence breamore...@yahoo.co.uk wrote: On 24/01/2014 16:37, Jesse Noller wrote: fwiw, I'm offering the keys/account/etc for getpython3.com to whomever has the time to keep it fresh and up to date. If I've ever heard of this I've forgotten about it. How do we make it more prominent? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 5:30 AM, M.-A. Lemburg m...@egenix.com wrote: On 22.01.2014 11:56, Donald Stufft wrote: On Jan 22, 2014, at 5:51 AM, M.-A. Lemburg m...@egenix.com wrote: On 22.01.2014 11:30, Donald Stufft wrote: I would like to propose that a backwards incompatible change be made to Python to make verification of hostname and certificate chain the default instead of requiring it to be opt in. Python 3.4 has made great strides in making it easier for applications to simply turn on these settings, however many people are not aware at all that they need to opt into this. Most assume that it will operate similarly to their browser, curl, wget, etc and validate by default and in the typical style of security related issues it will appear to work just fine however be grossly insecure. In the real world “opt in security” typically translates to just plain old insecure for the bulk of applications/libraries. I believe that Python has a responsibility to do the right thing by default here and it is in the best position to do so. The alternative requires every Python developer who wants to access a secure resource to be educated on the fact that they need to flip some switch to do what most of them would expect. Such a change would introduce considerable breakage. This would either have to be done using our usual pending deprecation, deprecation, removal dance (over three releases) or be postponed until Python 4. I can understand the need for doing the typical deprecation dance, although I believe such policies are often overlooked or accelerated for security sensitive changes. I do believe that waiting until Python 4 would be doing a great disservice to the users of Python though. Well, it's not really a security issue, since the security features are present in Python 3.4. It's just that the user has to enable them. I have to concur with Donald here - in the case of security, especially language security which directly impacts the implicit security of downstream applications, I should not have to opt in to the most secure defaults. Yes; this potentially breaks applications relying on insecure / loose defaults. However it changes the model to you are by default, explicitly secure then relying on the domain knowledge of an application developer to harden their application. When, if this changes, an application breaks, it will be in a plainly obvious way which can quickly be resolved. Donald is perfectly right: today, it's trivial to MITM an application that relies off of the current behavior; this is bad news bears for users and developers as it means they need domain knowledge to secure their applications by default they may not have. Note that several python.org services use CAcerts which would no longer be accessible per default following such a change. Not that it has much to do with this proposal, but those services should be switched to use certificates that are well trusted. Oh, it does have to do with the proposal, since it's a prominent example of services that would no longer work. While we can potentially change the situation for python.org servers, you have the same issues with other services which we don't have any influence on. The change would also disable all services using self-signed certificates which are very common in internal networks and for ad-hoc setups. Many routers and other devices use self-signed certificates when offering HTTPS services. I think overall, it's good to have default security, but locking out all certificates which do not have their root CA certs installed in default installations of systems per default would likely lead to people seeking other more insecure ways of getting things to work, rather than asking their admins to add their CA certs to the certificate chain configuration. So I'm not sure whether raising errors is the best way to achieve better default security. Perhaps just using warnings would be better. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 6:58 AM, Chris Angelico ros...@gmail.com wrote: On Wed, Jan 22, 2014 at 11:15 PM, Donald Stufft don...@stufft.io wrote: Do you really think those people would be making the same complaints if they could restore the previous behavior with a simple boolean flag delivered either via environment variable or in their own code? You assume that it's easy to tweak the code. From personal experience just today I can say that this isn't always the case. I was asked a question about an internal program that had been in use since the late 1990s, and which had originally been written to work with Netscape Navigator and had been updated to work with Firefox, but not Chrome. The original author is still around, but it's too much hassle to get that code dug into, so it's far easier just to accept a small issue with Chrome (since the program's not used very often anyway). But if Chrome had completely broken that program, the solution would simply be keep using Firefox, not fix the program - it's not considered a bug. Now, maybe it wouldn't be a problem if the fix is an environment variable, but imagine a thousand-computer deployment and you have to tweak the environment on all of them. Feel like doing that just because the newest Python needs it? Not so much. What's the bet that that application will be ported to python 3.4/3.5 if this is the case? I'd say approaching 0, which is ok. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Jan 22, 2014, at 8:03 AM, Christian Heimes christ...@python.org wrote: On 22.01.2014 14:55, Donald Stufft wrote: As an additional side note, anecdotal evidence and what not, but *every* time I bring this up somewhere I get at least one reply that looks similar to https://twitter.com/ojiidotch/status/425986619879866368 Yeah :( The ssl module documentation http://docs.python.org/3/library/ssl.html features a big red warning box for a good reason. And no one reads it. I can't count the number of times I've gotten called into a managers office when they find out python doesn't do cert validation by default (and in 2, it's not been trivial) and gotten told to fix it, or we move off of python. Donald is perfectly right: every time you point out to users that this is the default behavior the response is almost universally you can't be serious, is this a joke? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On Wed, Jan 22, 2014 at 3:48 PM, Donald Stufft don...@stufft.io wrote: Never mind. If someone else cares they can propose it. I withdraw. I'll throw writing a PEP for 3.5 to do this following the deprecation policy on my todo list so 3.4 fixing can move on. I needed to brush up on my ReST anyway. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Getting Tulip (PEP 3156) into the 3.4 stdlib, marked provisional, named asyncio
On Sep 28, 2013, at 12:59 AM, Nick Coghlan ncogh...@gmail.com wrote: It sounds like a reasonable approach to me. In terms of naming, would you consider concurrent.asyncio? When we created that parent namespace for futures, one of the other suggested submodules discussed was the standard event loop API. +1 but up to guido Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official github mirror for CPython?
Email me the name of the repo you want, and your github username (preferably off list so I don't miss it!) and I will set you up Eli On Fri, Sep 27, 2013 at 11:02 AM, Eli Bendersky eli...@gmail.com wrote: On Thu, Jul 25, 2013 at 7:48 AM, Brian Curtin br...@python.org wrote: On Thu, Jul 25, 2013 at 9:37 AM, Christian Heimes christ...@python.org wrote: Am 25.07.2013 16:29, schrieb Eli Bendersky: Hi all, I've been looking for a Github mirror for Python, and found two: * https://github.com/python-git/python has a lot of forks/watches/starts but seems to be very out of date (last updated 4 years ago) * https://github.com/python-mirror/python doesn't appear to be very popular but is updated daily Are some of you the owners of these repositories? Should we consolidate to a single semi-official mirror? +1 Does the PSF have an official account on github? We have one on bitbucket... I don't remember who runs this, and I thought I was in it (maybe just on BB), but: https://github.com/python The list of members for https://github.com/python is https://github.com/python?tab=members How can I get added to that list? When that happens, I can try to set-up a regularly updated CPython mirror. Eli ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Feb 27, 2013, at 3:20 PM, anatoly techtonik techto...@gmail.com wrote: * security by obscurity in legal position of PSF towards contributors https://code.google.com/legal/individual-cla-v1.0.html vs http://www.python.org/psf/contrib/contrib-form/ + http://www.samurajdata.se/opensource/mirror/licenses/afl-2.1.php or http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/licenses/apache2.0.php and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0 license I'm not even touching your security through obscurity trollbait, but I'd like to know how the PSF / Python core is non compliant with clause 4, and the legal counsel you spoke to to confirm this. Jesse___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Feb 28, 2013, at 6:42 AM, Stefan Krah ste...@bytereef.org wrote: Jesse Noller jnol...@gmail.com wrote: http://www.python.org/psf/contrib/contrib-form/ + http://opensource.org/ licenses/apache2.0.php and why PSF doesn't comply the 4. Redistribution clause from Apache 2.0 license I'm not even touching your security through obscurity trollbait, but I'd like to know how the PSF / Python core is non compliant with clause 4, and the legal counsel you spoke to to confirm this. Perhaps it's an idea to have a python-legal mailing list for these topics? I don't think it's fundamentally wrong to scrutinize licenses, provided that the discussion stays civil and factual. IIRC Debian has such a list because people got annoyed with the traffic on other lists. Stefan Krah We have one: p...@python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Feb 28, 2013, at 6:55 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 28 Feb 2013 06:48:24 -0500, Jesse Noller jnol...@gmail.com a écrit : Perhaps it's an idea to have a python-legal mailing list for these topics? I don't think it's fundamentally wrong to scrutinize licenses, provided that the discussion stays civil and factual. IIRC Debian has such a list because people got annoyed with the traffic on other lists. Stefan Krah We have one: p...@python.org That's not exactly a public mailing-list. Regards Antoine. Nope. But it's also where lawyers flock and these issues can rapidly be resolved. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Feb 28, 2013, at 7:23 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 28 Feb 2013 06:57:36 -0500, Jesse Noller jnol...@gmail.com a écrit : On Feb 28, 2013, at 6:55 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 28 Feb 2013 06:48:24 -0500, Jesse Noller jnol...@gmail.com a écrit : Perhaps it's an idea to have a python-legal mailing list for these topics? I don't think it's fundamentally wrong to scrutinize licenses, provided that the discussion stays civil and factual. IIRC Debian has such a list because people got annoyed with the traffic on other lists. Stefan Krah We have one: p...@python.org That's not exactly a public mailing-list. Regards Antoine. Nope. But it's also where lawyers flock and these issues can rapidly be resolved. Really? I didn't know lawyers flocked at the PSF. It also doesn't provide any public feedback for said resolution, meaning it doesn't help alleviate any future discussions about legal issues. Regards Antoine. Yes sir. Ill have a new mailing list made today. I will note on the list page that is is strictly unofficial and all legal / lawyer type things must be sent to p...@python.org for official decisions or legal counsel. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Feb 28, 2013, at 7:31 AM, Jesse Noller jnol...@gmail.com wrote: On Feb 28, 2013, at 7:23 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 28 Feb 2013 06:57:36 -0500, Jesse Noller jnol...@gmail.com a écrit : On Feb 28, 2013, at 6:55 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 28 Feb 2013 06:48:24 -0500, Jesse Noller jnol...@gmail.com a écrit : Perhaps it's an idea to have a python-legal mailing list for these topics? I don't think it's fundamentally wrong to scrutinize licenses, provided that the discussion stays civil and factual. IIRC Debian has such a list because people got annoyed with the traffic on other lists. Stefan Krah We have one: p...@python.org That's not exactly a public mailing-list. Regards Antoine. Nope. But it's also where lawyers flock and these issues can rapidly be resolved. Really? I didn't know lawyers flocked at the PSF. It also doesn't provide any public feedback for said resolution, meaning it doesn't help alleviate any future discussions about legal issues. Regards Antoine. Yes sir. Ill have a new mailing list made today. I will note on the list page that is is strictly unofficial and all legal / lawyer type things must be sent to p...@python.org for official decisions or legal counsel. Spent a minute driving to request the new list. Will notify when it's up ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Feb 28, 2013, at 8:03 AM, Stephen J. Turnbull step...@xemacs.org wrote: Stefan Krah writes: Why would [the PSF list] help to resolve such an issue (if it is an issue at all!) Precisely. If there *is* a compliance problem, there's nothing to be done before talking to the lawyers. Although license *choice* is primarily a political issue, *compliance* is technical. And the board is legally bound to ensure compliance. I know, I've gotten the email requests clarifying copyright assignments. That being said, I requested a mailing list where this can all be hotly debated and will let everyone know when the playground is open. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] New mailing list: Python-legal-sig
See: http://mail.python.org/mailman/listinfo/python-legal-sig Open archives. As the header says this is for the discussion of CLA/other issues. If specific legal questions or alterations to Python/the PSF trademarks, CLA/etc are requested those *must* be sent to p...@python.org for board oversight and approval. This is an open discussion list, and is not official legal stance of the foundation, trademarks, etc. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Thursday, February 28, 2013 at 7:26 AM, Stefan Krah wrote: Jesse Noller jnol...@gmail.com (mailto:jnol...@gmail.com) wrote: We have one: p...@python.org (mailto:p...@python.org) That's not exactly a public mailing-list. Nope. But it's also where lawyers flock and these issues can rapidly be resolved. If the list isn't publicly archived, the same questions will arise over and over again (probably on python-dev). Why would it help to resolve such an issue (if it is an issue at all!) for a single person on a private mailing list? See: http://mail.python.org/pipermail/python-dev/2013-February/124463.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Thursday, February 28, 2013 at 9:49 AM, Stefan Krah wrote: Jesse Noller jnol...@gmail.com (mailto:jnol...@gmail.com) wrote: Why would it help to resolve such an issue (if it is an issue at all!) for a single person on a private mailing list? See: http://mail.python.org/pipermail/python-dev/2013-February/124463.html That was quick. Thanks! Stefan Krah It only takes a minute or two to do something good. Now how about dem XML vulnerabilities? ;) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Feb 21, 2013, at 5:32 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 21 Feb 2013 11:18:35 +0100, Christian Heimes christ...@python.org a écrit : Am 21.02.2013 08:42, schrieb Antoine Pitrou: Sure, but in many instances, rebooting a machine is not business-threatening. You will have a couple of minutes' downtime and that's all. Which is why the attack must be repeated many times to be a major annoyance. Is this business-threatening enough? https://pypi.python.org/pypi/defusedxml#external-entity-expansion-remote You haven't proved that these were actual threats, nor how they actually worked. I'm gonna remain skeptical if there isn't anything more precise than It highly depends on the parser and the application what kind of exploit is possible. Regards Antoine. I guess someone need to write a proof of concept exploit for you and release it into the wild. Ok ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XML DoS vulnerabilities and exploits in Python
On Feb 20, 2013, at 6:22 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 20 Feb 2013 18:21:22 -0500 Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 20, 2013 at 6:08 PM, Antoine Pitrou wrote: It's not a distributed DoS issue, it's a severe DoS vulnerabilities. A single 1 kB XML document can kill virtually any machine, even servers with more than hundred GB RAM. Assuming an attacker can inject arbitrary XML. Not every XML document is loaded from the Internet. Even documents not loaded from the internet can be at risk. Often times security breaches are the result of a chain of actions. You can say I'm not loading this XML from the internet, so therefore I am safe but then you have another flaw (for example) where you unpack a zip file without verifying there are not absolute paths and suddenly your xml file has been replaces with a malicious one. Assuming your ZIP file is coming from the untrusted Internet, indeed. Again, this is the same assumption that you are grabbing some important data from someone you can't trust. Just because you are living in a Web-centric world doesn't mean everyone does. There are a lot of use cases which are not impacted by your security rules. Bugfix releases shouldn't break those use cases, which means the security features should be mostly opt-in for 2.7 and 3.3. Regards Antoine. Any type of input is a potential attack vector; this isn't web centric, it's a systemic flaw in the spec that allows any application that's loading XML to be bombed into oblivion. People need to trust that the standard library is reliable and sane-by-default. What we have right now isn't ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My CLA
On Monday, February 11, 2013 at 2:23 PM, Antoine Pitrou wrote: On Mon, 11 Feb 2013 22:07:50 +0300 anatoly techtonik techto...@gmail.com (mailto:techto...@gmail.com) wrote: On Mon, Feb 11, 2013 at 9:27 PM, Guido van Rossum gu...@python.org (mailto:gu...@python.org) wrote: Anatoly, stop this discussion *NOW*. It is not appropriate for python-dev and you risk being banned from python-dev if you keep it up. It is not a problem for me to keep silence for another couple of months. But this weekend there will be an open source conference in Belarus [1], and I will need to explain what this specific CLA is about in developer-friendly language translated to Russian. The Python contributor agreement allows the PSF to safely redistribute your contributions under its own license, the PSF license. The Python contributor agreement is *not* a copyright assignment: you legally remain the author of the code you contributed (i.e. you can also publish it elsewhere under any license you want). Regards Antoine. FWIW: Django's FAQ spells out the same reasons we have one for Python: https://www.djangoproject.com/foundation/cla/faq/ just s/Django/Python/ and s/Django Software Foundation/Python Software Foundation/ - it's a good concise FAQ. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3156 - Asynchronous IO Support Rebooted
On Friday, December 21, 2012 at 1:57 PM, Guido van Rossum wrote: Dear python-dev *and* python-ideas, I am posting PEP 3156 here for early review and discussion. As you can see from the liberally sprinkled TBD entries it is not done, but I am about to disappear on vacation for a few weeks and I am reasonably happy with the state of things so far. (Of course feedback may change this. :-) Also, there has already been some discussion on python-ideas (and even on Twitter) so I don't want python-dev to feel out of the loop -- this *is* a proposal for a new standard library module. (But no, I haven't picked the module name yet. :-) There's an -- also incomplete -- reference implementation at http://code.google.com/p/tulip/ -- unlike the first version of tulip, this version actually has (some) unittests. Let the bikeshedding begin! (Oh, happy holidays too. :-) -- --Guido van Rossum (python.org/~guido (http://python.org/~guido)) I really do like tulip as the name. It's quite pretty. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Installation on Macs
I think becoming an apple signed developer to get a cert is the best approach. If anyone wanted to approach apple about open source/non profit gratis licenses, that would be appreciated. Otherwise I could do it / fund it from the PSF board side, which I am happy to do. I also concur with Raymond that the download/install instructions could be simplified. Noting for users that rather than downloading Xcode, they can just download the OSX Command Line Tools installer and easy_install/pip/etc will just work would also be nice Jesse On Aug 14, 2012, at 8:33 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Mountain Lion, the default security settings only allow installation of applications downloaded from the Mac App Stored and identified developers. We need to either become an identified developer or include some instructions on how to change the security settings (System Preference -- General -- Unlock --Select the Anywhere radio button -- Install Python -- Restore the original settings -- and Relock). Changing the security settings isn't appealing because 1) it weakens the user's security 2) it involves multiple steps and 3) the user will see an unsettling warnings along the way. Another unrelated issue is that the instructions for updating Tcl/Tk are problematic. In the past few months, I've witnessed hundreds of people unsuccessfully trying follow the instructions and having an immediate unpleasant out-of-the-box experience when IDLE crashes. I suggest that we stop being so indirect about the chain of footnotes and links leading to a Tcl/Tk download. I would like to see a direct Tcl/Tk updater link side-by-side with our Python installer link at http://www.python.org/download/releases/2.7.3/ Someone did add a note the the IDLE startup screen to the effect of: WARNING: The version of Tcl/Tk (8.5.9) in use may be unstable. Visit http://www.python.org/download/mac/tcltk/ for current information. In some ways this is progress. In others, it falls short. If IDLE crashes, you can't see the message. If you have installed the ActiveTCL 8.5.12 update, you still see the warning eventhough it isn't necessary. Also, I don't link that the referenced page is so complex and that it is full unsettling warnings, important notices, do-not-use advice, mentions of instability, etc. I would like to see our download page have something more simple, affirmative, positively worded and direct. For example: * Mac OS X 64-bit/32-bit Installer (3.2.3) for Mac OS X 10.6 and 10.7 [2] (sig). To run IDLE or Tkinter, you need to update Tcl/Tk to ActiveTcl 8.5.12 http://www.activestate.com/activetcl/downloads . That saves you from having to click a links down to a footnote at the bottom of the page http://www.python.org/download/releases/2.7.3/#id6 that sends you to http://www.python.org/download/mac/tcltk which is another page full of tables, warnings,etc that leads you to the Apple 8.5.9 section http://www.python.org/download/mac/tcltk/#apple-8-5-9 which is a dead-end because there are still known issues with 8.5.9, leaving you with the ActiveTCL section http://www.python.org/download/mac/tcltk/#activetcl-8-5-12 which has a paragraph of text obscuring the link you actually needed: http://www.activestate.com/activetcl/downloads . I applaud that some effort was made to document a solution; however, in practice the daisy chain of footnotes, tables, and links has proven unworkable for most of the engineers I've been working with. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
More fuel; fire: http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] VS 11 Express is Metro only.
On Friday, June 8, 2012 at 3:01 PM, Meador Inge wrote: On Fri, May 25, 2012 at 7:06 AM, mar...@v.loewis.de (mailto:mar...@v.loewis.de) wrote: I hereby predict that Microsoft will revert this decision, and that VS Express 11 will be able to build CPython. And your prediction was right on :-) : http://blogs.msdn.com/b/visualstudio/archive/2012/06/08/visual-studio-express-2012-for-windows-desktop.aspx. Martin's predictions are sometimes eerily correct. ;) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] regarding HTML mail
I'd like to discuss top-posting. On Monday, March 19, 2012 at 2:20 PM, Barry Warsaw wrote: On Mar 19, 2012, at 05:25 PM, Tshepang Lekhonkhobe wrote: apology: I searched for a few minutes and could not find a code of conduct regarding HTML mail. I'm not sure it's written down anywhere, but in general we strongly discourage HTML email for the lists @python.org (http://python.org) Can we have some guideline to allow only plain text emails, so as to avoid cases like http://mail.python.org/pipermail/docs/2012-March/007999.html, where you are forced to scroll horizontally in order to read the text. docs is a different mailing list than python-dev, but still neither list is doing any content filtering. We could always enable that if we wanted to get strict about it. In this case, this isn't html email, it's likely this bug: https://bugs.launchpad.net/mailman/+bug/558294 Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org (mailto:Python-Dev@python.org) http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Spreading the Python 3 religion (was Re: PEP 414 - Unicode Literals for Python 3)
FWIW, I agree that much of the rhetoric in the current version of PEP 414 is excessive. Armin has given me permission to create an updated version of PEP 414 and toning down the hyperbole (or removing it entirely in cases where it's irrelevant to the final decision) is one of the things that I will be changing. I also plan to add a link to Lennart's guide to the various porting strategies that are currently available, more clearly articulate the cases where the new approach can most help (i.e. when there are project specific reasons to avoid the unicode_literals import), as well as name drop Pyramid (Chris McDonough), Flask (Armin), Django (Jacob Kaplan-Moss) and requests (Kenneth Reitz) as cases where key developers of web-related third party frameworks or libraries have indicated that PEP 414 will help greatly with bringing the sections of the Python ecosystem they're involved with into the Python 3 fold over the next few years. My aim is for the end result to better reflect the reasons why Guido *accepted* the PEP, moreso than Armin's own reasons for *wanting* it. Thank you Nick and Armin. I think toning down the rhetoric is a very amicable solution. Let me know if I need to add anything to http://getpython3.com/ (have linked porting guides there too if you want) jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On Saturday, February 25, 2012 at 10:13 PM, Guido van Rossum wrote: If this can encourage more projects to support Python 3 (even if it's only 3.3 and later) and hence improve adoption of Python 3, I'm all for it. A small quibble: I'd like to see a benchmark of a 'u' function implemented in C. --Guido After having this explained quite a bit to me by the more web-savvy folks such as Armin and Chris M/etc, I am a +1, the rationale makes sense, and much for the same reason that Guido cites, I think this will help with code bases using the single code base approach, and assist with overall adoption. +1 jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] http://pythonmentors.com/
I've been trying to publicize it on twitter, my blog, google plus and elsewhere. help welcome. On Friday, February 10, 2012 at 8:27 PM, Mark Lawrence wrote: Hi all, I'd never heard of this until some Dutch geezer whose name I'm now forgotten pointed me to it. Had I known about it a couple of years ago it would have saved a lot of people a lot of grief. Please could it be given a bit of publicity. -- Cheers. Mark Lawrence. p.s. The Dutch geezer in question competes with Dr. Who for having the bast time travelling machine :) ___ Python-Dev mailing list Python-Dev@python.org (mailto:Python-Dev@python.org) http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Wednesday, December 28, 2011 at 8:28 PM, Michael Foord wrote: Hello all, A paper (well, presentation) has been published highlighting security problems with the hashing algorithm (exploiting collisions) in many programming languages Python included: http://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf Although it's a security issue I'm posting it here because it is now public and seems important. The issue they report can cause (for example) handling an http post to consume horrible amounts of cpu. For Python the figures they quoted: reasonable-sized attack strings only for 32 bits Plone has max. POST size of 1 MB 7 minutes of CPU usage for a 1 MB request ~20 kbits/s → keep one Core Duo core busy This was apparently reported to the security list, but hasn't been responded to beyond an acknowledgement on November 24th (the original report didn't make it onto the security list because it was held in a moderation queue). The same vulnerability was reported against various languages and web frameworks, and is already fixed in some of them. Their recommended fix is to randomize the hash function. All the best, Michael Back up link for the PDF: http://dl.dropbox.com/u/1374/2007_28C3_Effective_DoS_on_web_application_platforms.pdf Ocert disclosure: http://www.ocert.org/advisories/ocert-2011-003.html jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
On Wednesday, December 28, 2011 at 8:37 PM, Jesse Noller wrote: On Wednesday, December 28, 2011 at 8:28 PM, Michael Foord wrote: Hello all, A paper (well, presentation) has been published highlighting security problems with the hashing algorithm (exploiting collisions) in many programming languages Python included: http://events.ccc.de/congress/2011/Fahrplan/attachments/2007_28C3_Effective_DoS_on_web_application_platforms.pdf Although it's a security issue I'm posting it here because it is now public and seems important. The issue they report can cause (for example) handling an http post to consume horrible amounts of cpu. For Python the figures they quoted: reasonable-sized attack strings only for 32 bits Plone has max. POST size of 1 MB 7 minutes of CPU usage for a 1 MB request ~20 kbits/s → keep one Core Duo core busy This was apparently reported to the security list, but hasn't been responded to beyond an acknowledgement on November 24th (the original report didn't make it onto the security list because it was held in a moderation queue). The same vulnerability was reported against various languages and web frameworks, and is already fixed in some of them. Their recommended fix is to randomize the hash function. All the best, Michael Back up link for the PDF: http://dl.dropbox.com/u/1374/2007_28C3_Effective_DoS_on_web_application_platforms.pdf Ocert disclosure: http://www.ocert.org/advisories/ocert-2011-003.html And more analysis/information: http://cryptanalysis.eu/blog/2011/12/28/effective-dos-attacks-against-web-application-plattforms-hashdos/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PATCH] Adding braces to __future__
On Friday, December 9, 2011 at 3:26 PM, Cedric Sodhi wrote: IF YOU THINK YOU MUST REPLY SOMETHING WITTY, ITERATE THAT THIS HAD BEEN DISCUSSED BEFORE, REPLY THAT IT'S SIMPLY NOT GO'NNA HAPPEN, THAT WHO DOESN'T LIKE IT IS FREE TO CHOOSE ANOTHER LANGUAGE OR SOMETHING SIMILAR, JUST DON'T. Otherwise, read on. I know very well that this topic has been discussed before. On forums. Mailing lists. IRC. Blogs. From person to person, even. And I know equally well, from all those years experiencing argument-turned-debates on the internet, how a (minor|major) fraction of participants make up for their inability to lead a proper debate by speaking the loudest of all, so that eventually quantity triumphs over quality and logic. That ahead; I hope you can try not to fall in that category. Let instead reason prevail over sentimentalism, mislead purism, elitism, and all other sorts of isms which hinder advancement in the greater context. Python has surprised once already: The changes from 2 to 3 were not downwards compatible because the core developers realized there is more to a sustainable language than constantly patching it up until it comes apart like the roman empire. Let's keep that spirit for a second and let us discuss braces, again, with the clear goal of improving the language. End of disclaimer? End of disclaimer! Whitespace-Blocking (WSB) as opposed to Delimiter-Blocking (DB) has reasons. What are those reasons? Well, primarily, it forces the programmer to maintain well readable code. Then, some might argue, it is quicker to type. Two reasons, but of what importance are they? And are they actually reasons? You may guessed it from the questions themselves that I'm about to question that. I don't intend to connote brazen implications, so let me spell out what I just implied: I think anyone who thinks that exclusive WSB is a good alternative or even preferable to DB is actually deluding themselves for some personal version of one of those isms mentioned above. Let's examine these alleged advantages objectively one for one. But before that, just to calm troubled waters a little, allow me bring forward the conclusion: Absolutely no intentions to remowe WSB from Python. Although one might have gotten that impression from the early paragraphs, no intentions to break downwards compatibility, either. What Python needs is an alternative to WSB and can stay Python by still offering WSB to all those who happen to like it. Readable code, is it really an advantage? Two linebreaks, just for the suspense, then: Of course it is. Forcing the programmer to write readable code, is that an advantage? No suspense, the answer is Of course not. Python may have started off as the casual scripting language for casual people. People, who may not even have known programming. And perhaps it has made sense to force -- or shall we say motivate, since you can still produce perfectly obfuscated code with Python -- them to write readably. But Python has matured and so has its clientele. Python does not become a better language, neither for beginners nor for experienced programmers who also frequently use Python these days, by patronizing them and restricting them in their freedom. Readable code? Yes. Forcing people to write readable code by artificial means? No. Practice is evidence for the mischief of this policy: Does the FOSS community suffer from a notorious lack of proper indention or readability of code? Of course we don't. I'm not a native speaker, but dict.cc (http://dict.cc) tells me that what we call mit Kanonen auf Spatzen schießen (firing cannons at sparrows) is called breaking a fly on the wheel in English. I may lack the analogy for the fly on the wheel, which, if I'm not mistaken, used to be a device for torture in the Middle Ages, but I can tell you that the cannon ball which might have struck the sparrows, coincidently caused havoc in the hinterlands. For the wide-spread and professional language Python is today, the idea of forcing people to indent is misguided. These days, it may address a neglible minority of absolute beginners who barely started programming and would not listen to the simple advice of indenting properly, but on the other hand it hurts/annoys/deters a great community of typical programmers for whom DB has long become a de facto standard. For them, it's not a mere inconsistency without, for them, any apparent reason. It's more than the inconvenience not being able to follow ones long time practices, using the scripts one wrote for delimiters, the shortcuts that are usually offered by editor, etc. It also brings about a whole class of new problems which may be anticipated and prevent, yet bear a great potential for new, even hard-to-find bugs (just in case anyone would respond that we had eventually successfully redeemed the mismatched
Re: [Python-Dev] Python 3 optimizations continued...
On Aug 30, 2011, at 9:05 AM, Nick Coghlan ncogh...@gmail.com wrote: On Tue, Aug 30, 2011 at 9:38 PM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 30 Aug 2011 13:29:59 +1000 Nick Coghlan ncogh...@gmail.com wrote: Anecdotal, non-reproducible performance figures are *not* the way to go about serious optimisation efforts. What about anecdotal *and* reproducible performance figures? :) I may be half-joking, but we already have a set of py3k-compatible benchmarks and, besides, sometimes a timeit invocation gives a good idea of whether an approach is fruitful or not. While a permanent public reference with historical tracking of performance figures is even better, let's not freeze everything until it's ready. (for example, do we need to wait for speed.python.org before PEP 393 is accepted?) Yeah, I'd neglected the idea of just running perf.py for pre- and post-patch performance comparisons. You're right that that can generate sufficient info to make a well-informed decision. I'd still really like it if some of the people advocating that we care about CPython performance actually volunteered to spearhead the effort to get speed.python.org up and running, though. As far as I know, the hardware's spinning idly waiting to be given work to do :P Cheers, Nick. Discussion of speed.python.org should happen on the mailing list for that project if possible. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
2011/8/29 Charles-François Natali neolo...@free.fr: +3 (agreed to Jesse, Antoine and Ask here). The http://bugs.python.org/issue8713 described non-fork implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing. I see two drawbacks: - it will be slower, since the interpreter startup time is non-negligible (well, normally you shouldn't spawn a new process for every item, but it should be noted) Yes; but spawning and forking are both slow to begin with - it's documented (I hope heavily enough) that you should spawn multiprocessing children early, and keep them around instead of constantly creating/destroying them. - it'll consume more memory, since we lose the COW advantage (even though it's already limited by the fact that even treating a variable read-only can trigger an incref, as was noted in a previous thread) cf Yes, it would consume slightly more memory; but the benefits - making it consistent across *all* platforms with the *same* restrictions gets us closer to the principle of least surprise. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Mon, Aug 29, 2011 at 1:16 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 29 Aug 2011 13:03:53 -0400 Jesse Noller jnol...@gmail.com wrote: 2011/8/29 Charles-François Natali neolo...@free.fr: +3 (agreed to Jesse, Antoine and Ask here). The http://bugs.python.org/issue8713 described non-fork implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing. I see two drawbacks: - it will be slower, since the interpreter startup time is non-negligible (well, normally you shouldn't spawn a new process for every item, but it should be noted) Yes; but spawning and forking are both slow to begin with - it's documented (I hope heavily enough) that you should spawn multiprocessing children early, and keep them around instead of constantly creating/destroying them. I think fork() is quite fast on modern systems (e.g. Linux). exec() is certainly slow, though. The third drawback is that you are limited to picklable objects when specifying the arguments for your child process. This can be annoying if, for example, you wanted to pass an OS resource. Regards Antoine. Yes, it is annoying; but again - this makes it more consistent with the windows implementation. I'd rather that restriction than the sanitization of the ability to use threading and multiprocessing alongside one another. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Mon, Aug 29, 2011 at 1:22 PM, Antoine Pitrou solip...@pitrou.net wrote: Le lundi 29 août 2011 à 13:23 -0400, Jesse Noller a écrit : Yes, it is annoying; but again - this makes it more consistent with the windows implementation. I'd rather that restriction than the sanitization of the ability to use threading and multiprocessing alongside one another. That sanitization is generally useful, though. For example if you want to use any I/O after a fork(). Oh! I don't disagree; I'm just against the removal of the ability to mix multiprocessing and threads; which it does internally and others do in every day code. The proposed removal of that functionality - using the two together - would leave users in the dust, and not needed if we patch http://bugs.python.org/issue8713 - which at it's core is just an addition flag. We could document the risk(s) of using the fork() mechanism which has to remain the default for some time. The point is, is that the solution to http://bugs.python.org/issue6721 should not be intertwined or cause a severe change in the multiprocessing module (e.g. rewriting from scratch), etc. I'm not arguing that both bugs should not be fixed. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Fri, Aug 26, 2011 at 3:18 AM, Nir Aides n...@winpdb.org wrote: Another face of the discussion is about whether to deprecate the mixing of the threading and processing modules and what to do about the multiprocessing module which is implemented with worker threads. There's a bug open - http://bugs.python.org/issue8713 which would offer non windows users the ability to avoid using fork() entirely, which would sidestep the problem outlined in the atfork() bug. Under windows, which has no fork() mechanism, we create a subprocess and then use pipes for intercommunication: nothing is inherited from the parent process except the state passed into the child. I think that deprecating the use of threads w/ multiprocessing - or at least crippling it is the wrong answer. Multiprocessing needs the helper threads it uses internally to manage queues, etc. Removing that ability would require a near-total rewrite, which is just a non-starter. I'd rather examine bug 8713 more closely, and offer this option for all users in 3.x and document the existing issues outlined in http://bugs.python.org/issue6721 for 2.x - the proposals in that bug are IMHO, out of bounds for a 2.x release. In essence; the issue here is multiprocessing's use of fork on unix without the following exec - which is what the windows implementation essentially does using subprocess. Adding the option to *not* fork changes the fundamental behavior on unix systems - but I fundamentally feel that it's a saner, and more consistent behavior for the module as a whole. So, I'd ask that we not talk about tearing out the ability to use MP and threads, or threads with MP - that would be crippling, and there's existing code in the wild (including multiprocessing itself) that uses this mix without issue - it's stripping out functionality for what is a surprising and painful edge case that rarely directly affects users. I would focus on the atfork() patch more directly, ignoring multiprocessing in the discussion, and focusing on the merits of gps' initial proposal and patch. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving forward with the concurrent package
On Wed, Aug 10, 2011 at 4:45 PM, Brian Curtin brian.cur...@gmail.com wrote: On Wed, Aug 10, 2011 at 15:36, Antoine Pitrou solip...@pitrou.net wrote: Le Wed, 10 Aug 2011 14:54:33 -0500, Benjamin Peterson benja...@python.org a écrit : 2011/8/10 Brian Curtin brian.cur...@gmail.com: Now that we have concurrent.futures, is there any plan for multiprocessing to follow suit? PEP 3148 mentions a hope to add or move things in the future Is there some sort of concrete proposal? The PEP just seems to mention it as an idea. In general, -1. I think we don't need to be moving things around more to little advantage. Agreed. Also, flat is better than nested. Whoever wants to populate the concurrent package should work on new features to be added to it, rather than plans to rename things around. I agree with flat being better than nested and won't be pushing to move things around, but the creation of the concurrent package seemed like a place to put those things. I just found myself typing concurrent.multiprocessing a minute ago, so I figured I'd put it out there. I would like to move certain *features* of multiprocessing into that namespace - some things like map and others don't belong in the multiprocessing namespace, and should have been put into concurrent.* a long time ago. As for my plans: I had intended on making multiprocessing a closer corollary to threading, and moving the bigger features that should have been broken out into a different package (such as http://bugs.python.org/issue12708) and the managers. Those plans are obviously stalled as my time is being spent elsewhere. I disagree on the flat is better than nested point - multiprocessing's namespace is flat - but bloated, and many of it's features could work just as well in a threaded context (e.g, they are generally useful outside of multiprocessing alone). Regardless; currently I can't lead this, and multiprocessing-sig is silent. Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Speed.Python.org
Now that we have the machine, we need to start working on collecting/organizing the resources needed to get a shared codespeed system in place. After speaking with various people, we felt that overloading codespeed-dev, pypy-dev or python-dev with the discussions around this would be sub optimal. I've spun up a new mailing list here: http://mail.python.org/mailman/listinfo/speed Those who are interested in working on or contributing to the speed.python.org project can subscribe there. I personally can not lead the project, and so I will be looking to the current speed.pypy.org team, and python-dev contributors for leadership in this. I got you the hardware and hosting! :) jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] speed.python.org machine online
I've posted a more expansive entry on my blog: http://jessenoller.com/2011/06/29/announcing-the-new-speed-python-org-machine/ But the short version, that as discussed at the VM and language summit, we now have a hosted machine dedicated to the running of cross-interpreter speed tests, etc. The hardware was generously donate by HP and the hosting provided, again, free, by OSU/OSL. DL380 HP DL380G7 X5670 LFF (2U) Dual HP NC382i Dual Port Multifunction Gigabit Server Adapters HP Smart Array P410i/1GB FBWC Controller 4x 4GB (1x4GB) Dual Rank x4 PC3-10600 (DDR3-1333) Registered CAS-9 Memory Kit 2x HP 750W Common Slot Gold Hot Plug Power Supply Kit HP iLO Advanced including 1yr 24x7 Technical Support and Updates Electronic License 4x HP 300GB 6G SAS 15K rpm LFF (3.5-inch) Dual Port Enterprise 3yr Warranty Hard Drive 2 HP DL380 G7 Intel® Xeon® X5680 (3.33GHz/6-core/130W/12MB) FIO Processor Kit With hyperthreading on, the machine has 24 cores, and handily translates pypy using cpython 2.7 in about half the time it typically takes. I am looking forward to handing this over to the team who will be running with the project from here on out - special thanks to Van, Bob Gobeille at HP and the entire OSU/OSL team. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] looking for a contact at Google on the Blogger team
On Mon, May 23, 2011 at 2:15 AM, Nick Coghlan ncogh...@gmail.com wrote: On Sat, May 21, 2011 at 7:47 AM, Martin v. Löwis mar...@v.loewis.de wrote: As Jesse has said, there is an RFP in development to improve python.org to the point where we can self-host blogs and the like and deal with the associated user account administration appropriately. To run a blog on www.python.org, a PEP is not needed. If anybody would volunteer to set this up, it could be done in no time. If I understand correctly, the RFP is more about improving the entire python.org toolchain to make it something that non-programmers can easily provide content for (and even *programmers* don't particularly like the current toolchain). Cheers, Nick. That is correct. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] looking for a contact at Google on the Blogger team
On Fri, May 20, 2011 at 5:39 AM, Eli Bendersky eli...@gmail.com wrote: With respect to Google Blogger, I don't see a good reason to use it as the platform for the blog. As with any infrastructure, there is a reasonably high cost in changing, as people have become used to a certain way of doing things, and porting the contents from the old system to the new one requires additional effort. Blogger has its problems, but it typically gets the job done well enough (modulo cases like the one currently affecting Doug and his team). Has the Python insider blog really accumulated enough history and cruft to make this move problematic? It's a fairly new blog, with not much content in it. From my blogging experience, Blogger has other limitations which eventually bite you, and since it's not very flexible you can either live with it or move to a more flexible platform. All of this completely IMHO, of course. Just friendly advice ;-) Eli There is ongoing work for an RFP by the board to improve the python.org publishing system/site to allow us to self-host these things. Moving PSF properties off of it, and onto another hosted by someone else site is probably not a good idea, but our hands may be forced if google/blogger can not resolve the issues. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On Tue, Apr 19, 2011 at 1:06 AM, Stefan Behnel stefan...@behnel.de wrote: [snip] This PEP has received a lengthy discussion by now, so here's why I think it's being fought so heavily by several CPython core developers, specifically those who have traditionally carried a large part of the optimisation load in the project. I think the whole point of this PEP is that, having agreed that a shared standard library for all Python implementations is a good thing, the amount of shareable code should be maximised. I doubt that anyone will argue against this goal. But that obviously includes all sides. If other implementations are free to cherry pick the targets of their own effort geared by the optimisation of their own implementation, and leave the whole burden of compatibility and code reusability on CPython, in addition to the CPython efforts of improving and optimising its own core code base and its own stdlib version, it's not an equal matter. I am going to go out on a limb here and state that once the stdlib is shared, it is all of the VM's responsibility to help maintaining it, meaning the PEP 399 is binding to all of the VMs. If Jython wants to write an accelerator module in Java for something in the stdlib, they have to follow the same guidelines, same applies to PyPy, etc. I think this is an equal matter, and if needed, we should make note of it in the PEP. The goal here is to make it easier to share the code base of the stdlib and not pull the rug out from other implementations by having a stdlib module written only in highly optimized C with no Python fallback, leaving them with the unsavory duty of reimplementing it in Python|Java|C#, etc. Pure Python is the coin of the realm. That's what makes the PEP feel so unfair to CPython developers, because they are the ones who carry most of the burden of maintaining the stdlib in the first place, and who will most likely continue to carry it, because other implementations will continue to be occupied with their own core development for another while or two. It is nice to read that other implementations are contributing back patches that simplify their own reuse of the stdlib code. However, that does not yet make them equal contributors to the development and the maintenance of the stdlib, and is of very little worth to the CPython project. It often even runs counter to the interest of CPython itself. Sure, at first glance this seems to place an unfair burden on CPython - because we're just as guilty as being closed to other implementation as the other implementations are to us. We're trying to change that, and someone (us, as the reference implementation) need to take the first responsible step. Once this move is made/accepted, I would expect the other implementation to rapidly move away from their custom implementations of the stdlib and contribute to the shared code base and documentation. Yes, this places a burden on CPython, but in the long term in benefits *all* of the projects equally by simply having more active contributors. We have over 200 stdlib modules, and far, far less than that in active developers focused or working on the stdlib. Making it a shared property (in theory) means that the other VMs have a shared interest in that property. We're effectively spreading the load. I think this social problem of the PEP can only be solved if the CPython project stops doing the major share of the stdlib maintenance, thus freeing its own developer capacities to focus on CPython related improvements and optimisations, just like the other implementations currently do. I'm not sure we want that at this point. That's not going to happen. CPython will continue to do the bulk of the maintenance until we break it out, and the other implementations have time to adapt and pull in the shared code base. I don't see this as such a large burden as you seem to be making it out to be: CPython is the reference implementation, and our stdlib is the reference stdlib. We can break out the stdlib and share it amongst the implementations therefore making it more than the reference stdlib - we can make it the defacto stdlib for the language as a whole. We also, long term, want spread the maintenance load beyond CPython, but right now we are the primary caretakers, so yes - this adds load to us in the short term, but benefits us in the long term. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On Tue, Apr 19, 2011 at 8:17 AM, Maciej Fijalkowski fij...@gmail.com wrote: On Tue, Apr 19, 2011 at 10:57 AM, Nick Coghlan ncogh...@gmail.com wrote: On Tue, Apr 19, 2011 at 3:06 PM, Stefan Behnel stefan...@behnel.de wrote: I think this social problem of the PEP can only be solved if the CPython project stops doing the major share of the stdlib maintenance, thus freeing its own developer capacities to focus on CPython related improvements and optimisations, just like the other implementations currently do. I'm not sure we want that at this point. We've made a start on that aspect by granting CPython access to several of the core developers on the other VMs. The idea being that they can update the pure Python versions of modules directly rather than having to wait for one of us to do it on their behalf. Of course, as Maciej pointed out, that is currently hindered by the fact that the other VMs aren't targeting 3.3 yet, and that's where the main CPython development is happening. We're also slightly hindered by the fact that not all of us got privilages so far (Antonio Cuni in particular). Yeah, I emailed him this morning, I dropped the ball on his commit bit post pycon due to email overload. I'm resolving it today. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Releases for recent security vulnerability
On Sun, Apr 17, 2011 at 7:48 AM, Antoine Pitrou solip...@pitrou.net wrote: On Sat, 16 Apr 2011 21:32:48 -0500 Brian Curtin brian.cur...@gmail.com wrote: Three weeks after this security vulnerability was *publicly* reported on bugs.python.org, and two days after it was semi-officially announced, I'm still waiting for security updates for my Ubuntu and Debian systems! I reckon if this had been handled differently (i.e., making new releases and communicating it via the relevant channels [1]), we wouldn't have the situation we have right now. I don't really think there's a situation here, and I fail to see how the development blog isn't one of the relevant channels. If we want to make official announcements (like releases or security warnings), I don't think the blog is appropriate. A separate announcement channel (mailing-list or newsgroup) would be better, where people can subscribe knowing they will only get a couple of e-mails a year. Regards Antoine. And whose responsibility is it to email yet another mythical list? The person posting the fix? The person who found and filed the CVE? The release manager? Brian *helped* us by raising awareness of the issue: At least now there's a chance that one or more of the OS vendors *saw* that this was an issue that was fixed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Releases for recent security vulnerability
On Sun, Apr 17, 2011 at 9:42 AM, Antoine Pitrou solip...@pitrou.net wrote: Le dimanche 17 avril 2011 à 09:30 -0400, Jesse Noller a écrit : If we want to make official announcements (like releases or security warnings), I don't think the blog is appropriate. A separate announcement channel (mailing-list or newsgroup) would be better, where people can subscribe knowing they will only get a couple of e-mails a year. Regards Antoine. And whose responsibility is it to email yet another mythical list? The person posting the fix? The person who found and filed the CVE? The release manager? Well, whose responsibility is it to make blog posts about security issues? If you can answer this question then the other question shouldn't be any more difficult to answer ;) I don't think the people who may be interested in security announcements want to monitor a generic development blog, since Python is far from the only piece of software they rely on. /I/ certainly wouldn't want to. Also, I think Gustavo's whole point is that if we don't have a well-defined, deterministic procedure for security announcements and releases, then it's just as though we didn't care about security at all. Saying look, we mentioned this one on our development blog isn't really reassuring for the target group of people. Regards Antoine. I'm not arguing against us having a well defined, deterministic procedure! We need one, for sure - I'm just defending Brian's actions as perfectly rational and reasonable. Without his post, that CVE would have been published, publicly available on other sites (CVE tracking sites, and hence on the radar for people looking to exploit it), and no one would be the wiser. At least it got *some* attention this way. Is it the right thing to do moving forward? Probably not - but do we have the people/person willing to head up defining the policy and procedure, and do we have the needed contacts in the OS vendors/3rd party distributors to notify them rapidly in the case of fixing something like this? A lag of several weeks from fixing a security issue to a source level release from us that OS vendors can run with is too slow honestly. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Releases for recent security vulnerability
On Fri, Apr 15, 2011 at 8:30 AM, Brian Curtin brian.cur...@gmail.com wrote: On Apr 15, 2011 3:46 AM, Gustavo Narea m...@gustavonarea.net wrote: Hi all, How come a description of how to exploit a security vulnerability comes before a release for said vulnerability? I'm talking about this: http://blog.python.org/2011/04/urllib-security-vulnerability-fixed.html My understanding is that the whole point of asking people not to report security vulnerability publicly was to allow time to release a fix. To me, the fix *was* released. Sure, no fancy installers were generated yet, but people who are susceptible to this issue 1) now know about it, and 2) have a way to patch their system *if needed*. If that's wrong, I apologize for writing the post too early. On top of that, it seems I didn't get all of the details right either, so apologies on that as well. The code is open source: Anyone watching the commits/list know that this issue was fixed. It's better to keep it in the public's eyes, so they know *something was fixed and they should patch* than to rely on people *not* watching these channels. Assume the bad guys already knew about the exploit: We have to spread the knowledge of the fix as far and as wide as we can so that people even know there is an issue, and that it was fixed. This applies to users and *vendors* as well. A blog post is good communication to our users. I have to side with Brian on this one. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Releases for recent security vulnerability
On Fri, Apr 15, 2011 at 8:59 AM, Antoine Pitrou solip...@pitrou.net wrote: On Fri, 15 Apr 2011 08:36:16 -0400 Jesse Noller jnol...@gmail.com wrote: On Fri, Apr 15, 2011 at 8:30 AM, Brian Curtin brian.cur...@gmail.com wrote: On Apr 15, 2011 3:46 AM, Gustavo Narea m...@gustavonarea.net wrote: Hi all, How come a description of how to exploit a security vulnerability comes before a release for said vulnerability? I'm talking about this: http://blog.python.org/2011/04/urllib-security-vulnerability-fixed.html My understanding is that the whole point of asking people not to report security vulnerability publicly was to allow time to release a fix. To me, the fix *was* released. Sure, no fancy installers were generated yet, but people who are susceptible to this issue 1) now know about it, and 2) have a way to patch their system *if needed*. If that's wrong, I apologize for writing the post too early. On top of that, it seems I didn't get all of the details right either, so apologies on that as well. The code is open source: Anyone watching the commits/list know that this issue was fixed. It's better to keep it in the public's eyes, so they know *something was fixed and they should patch* than to rely on people *not* watching these channels. Assume the bad guys already knew about the exploit: We have to spread the knowledge of the fix as far and as wide as we can so that people even know there is an issue, and that it was fixed. This applies to users and *vendors* as well. True. However, many open source projects take the habit of cutting a release when a hole is discovered and fixed. It depends how seriously they (and their users) take security. Of course, there are different kinds of security issues, more or less severe. I don't know how severe the above issue is. Relying on a vendor distribution (such as a Linux distro, or ActiveState) is hopefully enough to get these security updates in time without patching anything by hand. I don't think many people compile Python for production use, but many do use our Windows installers. Regards Antoine. Agreed; but all I'm defending is the post describing what, and how it was fixed. Hiding it until we get around to eventually cutting a release doesn't make the fix, or vulnerability go away. We need to issue a release *quickly* - and we need to notify all of our consumers quickly. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [GSoC] Developing a benchmark suite (for Python 3.x)
On Fri, Apr 8, 2011 at 8:51 AM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/07/2011 07:52 PM, Michael Foord wrote: Personally I think the Gsoc project should just take the pypy suite and run with that - bikeshedding about what benchmarks to include is going to make it hard to make progress. We can have fun with that discussion once we have the infrastructure and *some* good benchmarks in place (and the pypy ones are good ones). So I'm still with Jesse on this one. If there is any discussion phase as part of the Gsoc project it should be very strictly bounded by time. Somehow I missed seeing '[GSoC]' in the subject line (the blizzard of notification messages to the various GSoC specific lists must've snow-blinded me :). I'm fine with leaving Cython out-of-scope for the GSoC effort, just not for perf.python.org as a whole. We don't need a massive outstanding todo list for perf.python.org - we need to get the current speed.pypy.org stuff made more generic for the purposes we're aiming for and to get the hardware (on my plate) first. Then we can talk about expanding it. I'm just begging that we not add a bunch of stuff to a todo list for something that doesn't exist right now. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [GSoC] Developing a benchmark suite (for Python 3.x)
On Thu, Apr 7, 2011 at 3:54 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Daniel, Thanks for putting this together. I am a huge supporter of benchmarking efforts. My brief comment is below. On Wed, Apr 6, 2011 at 11:52 AM, DasIch dasdas...@googlemail.com wrote: 1. Definition of the benchmark suite. This will entail contacting developers of Python implementations (CPython, PyPy, IronPython and Jython), via discussion on the appropriate mailing lists. This might be achievable as part of this proposal. If you are reaching out to other projects at this stage, I think you should also be in touch with the Cython people (even if its 'implementation' sits on top of CPython). As a scientist/engineer what I care about is how Cython benchmarks to CPython. I believe that they have some ideas on benchmarking and have also explored this space. Their inclusion would be helpful to me thinking this GSoC successful at the end of the day (summer). Thanks for your consideration. Be Well Anthony Right now, we are talking about building speed.python.org to test the speed of python interpreters, over time, and alongside one another - cython *is not* an interpreter. Cython is out of scope for this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [GSoC] Developing a benchmark suite (for Python 3.x)
On Thu, Apr 7, 2011 at 7:52 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: On 08/04/2011 00:36, Anthony Scopatz wrote: On Thu, Apr 7, 2011 at 6:11 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: On 07/04/2011 22:41, Antoine Pitrou wrote: On Thu, 07 Apr 2011 17:32:24 -0400 Tres Seavertsea...@palladion.com wrote: Right now, we are talking about building speed.python.org to test the speed of python interpreters, over time, and alongside one another - cython *is not* an interpreter. Cython is out of scope for this. Why is it out of scope to use the benchmarks and test harness to answer questions like can we use Cython to provide optional optimizations for the stdlib? I can certainly see value in havng an objective way to compare the macro benchmark performance of a Cython-optimized CPython vs. a vanilla CPython, as well as vs. PyPY, Jython, or IronPython. Agreed. Assuming someone wants to take care of the Cython side of things, I don't think there's any reason to exclude it under the dubious reason that it's not an interpreter. (would you exclude Psyco, if it was still alive?) Well, sure - but within the scope of a GSOC project limiting it to core python seems like a more realistic goal. Adding cython later shouldn't be an issue if someone is willing to do the work. Jesse, I understand that we are talking about the benchmarks on speed.pypy.org. The current suite, and correct me if I am wrong, is completely written in pure python so that any of the 'interpreters' may run them. My point, which I stand by, was that during the initial phase (where benchmarks are defined) that the Cython crowd should have a voice. This should have an enriching effect on the whole benchmarking task since they have thought about this issue in a way that is largely orthogonal to the methods PyPy developed. I think it would be a mistake to leave Cython out of the scoping study. Personally I think the Gsoc project should just take the pypy suite and run with that - bikeshedding about what benchmarks to include is going to make it hard to make progress. We can have fun with that discussion once we have the infrastructure and *some* good benchmarks in place (and the pypy ones are good ones). So I'm still with Jesse on this one. If there is any discussion phase as part of the Gsoc project it should be very strictly bounded by time. What michael said: My goal is is to get speed.pypy.org ported to be able to be used by $N interpreters, for $Y sets of performance numbers. I'm trying to constrain the problem, and the initial deployment so we don't spend the next year meandering about. It should be sufficient to port the benchmarks from speed.pypy.org, and any deltas from http://hg.python.org/benchmarks/ to Python 3 and the framework that runs the tests to start. I don't care if we eventually run cython, psyco, parrot, etc. But the focus at the language summit, and the continued focus of me getting the hardware via the PSF to host this on performance/speed.python.org is tightly focused on the pypy, ironpython, jython and cpython interpreters. Let's just get our basics done first before we go all crazy with adding stuff :) jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Information about how cpython in benchmarked
On Wed, Mar 30, 2011 at 1:19 AM, Nick Stinemates nstinema...@gmail.com wrote: This is really great to hear and something I would be hugely interested in contributing to. Lurking has paid off :) Nick Once I get the machine in place, and the team engaged, I am sure they'll be looking for help. As it stands, the best place to start helping is by getting in conact with the speed.pypy.org team (the pypy folks) and look into the codespeed codebase. We have to start work on it now to get it ready to be more general use. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documenting the buildbots
On Wed, Mar 30, 2011 at 5:01 PM, Antoine Pitrou solip...@pitrou.net wrote: For the record, I added a page documenting our continuous integration setup at: http://docs.python.org/devguide/buildbots.html Regards Antoine. that's awesome. should we document how to donate/add a buildbot somewhere in the same section (or alongside)? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documenting the buildbots
On Wed, Mar 30, 2011 at 5:24 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 30 Mar 2011 17:14:10 -0400 Jesse Noller jnol...@gmail.com wrote: On Wed, Mar 30, 2011 at 5:01 PM, Antoine Pitrou solip...@pitrou.net wrote: For the record, I added a page documenting our continuous integration setup at: http://docs.python.org/devguide/buildbots.html Regards Antoine. that's awesome. should we document how to donate/add a buildbot somewhere in the same section (or alongside)? It's documented at http://wiki.python.org/moin/BuildBot. I think it's a bit outside the scope of the devguide. (perhaps we should have an infrastructure/sysadmin contribution guide) Not a bad idea. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Information about how cpython in benchmarked
On Tue, Mar 29, 2011 at 7:00 AM, Nick Coghlan ncogh...@gmail.com wrote: On Tue, Mar 29, 2011 at 8:01 PM, Tennessee Leeuwenburg tleeuwenb...@gmail.com wrote: PyPy maintains http://speed.pypy.org/, which provides very clear information about the relative performance of PyPy trunk against some version of cpython (presumably 2.6 or 2.7). I'm not aware of a similar site for cpython, but that could easily just be my ignorance speaking. My interest is that I'm looking at building a benchmarking solution at work. and I can't think of a better way to build something good and general than to try and write something that could potentially be released as open source and be useful to others. As such I thought that benchmarking cpython would be a great use case, but I want to find out as much as I can about how people currently go about benchmarking Python. Initially I'm just looking at CPU profiling since it's easiest. One of the points coming out of the VM summit at Pycon is actually that we want to create a shared benchmarking site for CPython, PyPy, Jython, IronPython (and possibly Stackless) under the python.org banner (either speed.python.org, or possibly performance.python.org, since we want to do memory profiling as well). speed.pypy.org will be the reference site for this, but Maciej indicated at the VM summit that the code that runs that site needs some improvements before it will really be up to the task of effectively benchmarking multiple targets. So, according to http://speed.pypy.org/about/, the place to start with your benchmarking system would probably be https://github.com/tobami/codespeed. Cheers, Nick. Essentially echoing what nick said. I'm currently working on getting the HW for this together. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python Core Mentorship program
Hello everyone: I wanted to take a moment to outline another idea which came out of PyCon 2011 this year from numerous sources - a Python Core Mentorship Program predicated on the idea that Python-Core, and Python as a whole would be served by further lowering the barrier to entry of contribution, and to provide a program to connect new programmers, students, women, and others to experienced Python-Core developers (Mentors). Brett's revamp of the Dev guide was part one of secret plan to get more people involved in python-core - this is another part, but I'm not sure of the numbering scheme. The mission of the Python Core Mentor Program is to provide an open and welcoming place to connect students, programmers - and anyone interested in contributing to the Python-Core development. This project is based on the idea that the best way to welcome new people into any project is a venue which connects them to mentors who can assist in guiding them through the contribution process, including discussions on lists such as python-dev, and python-ideas, the bug tracker, mercurial, code reviews, etc. Additionally, mentors will assist in something incredibly critical to maintain contributor interest: getting patches through the process and actually *committed*. We all know - not everyone who is mentor will have all the answers, so mentors also act as conduits to others who will have the answer. The project itself will (hopefully) be low in time-spent, and largely self-managing. We will start simple with a mailing list (core-mentors...@python.org) where mentors, and those who wish to be mentored or ask questions may do so. This mailing list will have a code of conduct which will help prevent flame wars, or other counterproductive discussions - a code of conduct also makes it clear to mentors what they''re agreeing to when they decide to participate. The new list will also have a closed, members-only archive. After consulting with other core developers, we believe it's easier to ask questions when you don't have to worry about Google picking up your words from a public archive. We want to make this list a resource for people to be able to get started, ask silly questions, and so on - our goal is to turn anyone who wishes to be into an active, sustainable committer to Python. Mentors will be asked to answer questions - but also assist people in need of help with discussions on the mailing lists and bug tracker (conversations on which could have become contentious or stressful) and generally to be advocates for the people being mentored. For example - if a person submits a patch to the tracker, the mentor list may help them through initial code reviews, or discussions with other core developers. The job is to act as an experienced proxy for them. The first step to this project is to ask for volunteer mentors - people who are willing to help answer questions on the list, and generally guide people as needed being as friendly and courteous and welcoming as possible. If you are interested in being a mentor - or have feedback about this plan in general, please feel free to reach out to me (jnol...@gmail.com) directly. My goal, once this is setup, is to have the project largely self-managing, with the PSF helping to market it to the community as a whole. Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Core Mentorship program
On Fri, Mar 25, 2011 at 9:04 AM, exar...@twistedmatrix.com wrote: On 12:03 pm, jnol...@gmail.com wrote: Hello everyone: The new list will also have a closed, members-only archive. After consulting with other core developers, we believe it's easier to ask questions when you don't have to worry about Google picking up your words from a public archive. Boggle. Jean-Paul I assume that means your in, or you hate that idea? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Core Mentorship program
On Fri, Mar 25, 2011 at 9:26 AM, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Mar 25, 2011 at 11:06 PM, Jesse Noller jnol...@gmail.com wrote: On Fri, Mar 25, 2011 at 9:04 AM, exar...@twistedmatrix.com wrote: On 12:03 pm, jnol...@gmail.com wrote: Hello everyone: The new list will also have a closed, members-only archive. After consulting with other core developers, we believe it's easier to ask questions when you don't have to worry about Google picking up your words from a public archive. Boggle. I assume that means your in, or you hate that idea? I'll note that one thing the mentors on that list will be doing is using the questions received as an indication of areas where the devguide and FAQ may need possible tweaks. One other thing I would hope to be able to do with the list is to try to stay in touch with new contributors that participate in sprints. Cheers, Nick. Violently agreed on both counts. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Core Mentorship program
On Fri, Mar 25, 2011 at 8:03 AM, Jesse Noller jnol...@gmail.com wrote: Hello everyone: I wanted to take a moment to outline another idea which came out of PyCon 2011 this year from numerous sources - a Python Core Mentorship Program predicated on the idea that Python-Core, and Python as a whole would be served by further lowering the barrier to entry of contribution, and to provide a program to connect new programmers, students, women, and others to experienced Python-Core developers (Mentors). Brett's revamp of the Dev guide was part one of secret plan to get more people involved in python-core - this is another part, but I'm not sure of the numbering scheme. The mission of the Python Core Mentor Program is to provide an open and welcoming place to connect students, programmers - and anyone interested in contributing to the Python-Core development. This project is based on the idea that the best way to welcome new people into any project is a venue which connects them to mentors who can assist in guiding them through the contribution process, including discussions on lists such as python-dev, and python-ideas, the bug tracker, mercurial, code reviews, etc. Additionally, mentors will assist in something incredibly critical to maintain contributor interest: getting patches through the process and actually *committed*. We all know - not everyone who is mentor will have all the answers, so mentors also act as conduits to others who will have the answer. The project itself will (hopefully) be low in time-spent, and largely self-managing. We will start simple with a mailing list (core-mentors...@python.org) where mentors, and those who wish to be mentored or ask questions may do so. This mailing list will have a code of conduct which will help prevent flame wars, or other counterproductive discussions - a code of conduct also makes it clear to mentors what they''re agreeing to when they decide to participate. The new list will also have a closed, members-only archive. After consulting with other core developers, we believe it's easier to ask questions when you don't have to worry about Google picking up your words from a public archive. We want to make this list a resource for people to be able to get started, ask silly questions, and so on - our goal is to turn anyone who wishes to be into an active, sustainable committer to Python. Mentors will be asked to answer questions - but also assist people in need of help with discussions on the mailing lists and bug tracker (conversations on which could have become contentious or stressful) and generally to be advocates for the people being mentored. For example - if a person submits a patch to the tracker, the mentor list may help them through initial code reviews, or discussions with other core developers. The job is to act as an experienced proxy for them. The first step to this project is to ask for volunteer mentors - people who are willing to help answer questions on the list, and generally guide people as needed being as friendly and courteous and welcoming as possible. If you are interested in being a mentor - or have feedback about this plan in general, please feel free to reach out to me (jnol...@gmail.com) directly. My goal, once this is setup, is to have the project largely self-managing, with the PSF helping to market it to the community as a whole. Jesse And the mailing list is up and running for those of you interested in helping: http://mail.python.org/mailman/listinfo/core-mentorship jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Core Mentorship program
On Fri, Mar 25, 2011 at 5:16 PM, Guido van Rossum gu...@python.org wrote: On Fri, Mar 25, 2011 at 1:57 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: If you don't want a specific party snooping the site, just block that specific party. Why make a walled garden that *nobody* outside can look into? That undermines the free exchange of information. Surely a forum specifically for mentorship will be more useful if outsiders can be directed to existing discussions, without needing to join the private club. This argument comes up repeatedly. Some people object on principle to all closed lists. Other people will not participate in a discussion (or will not speak their minds about certain topics) unless they have some sense that the list is closed. The former group will then argue that the closedness of the list is an illusion, that there are many ways to snoop on the list regardless. (And there are, technically speaking.) The latter group will still feel a different comfort level and argue that *socially* it is not the same thing at all. I propose to give it a rest. If you want to know what's going on there, just subscribe, nobody will stop you (and if they did there are plenty of public forums to complain). You will find soon enough that nothing unsavory is being discussed. There's an understanding that only people, not bots/indexers/etc., subscribe to the list, and that the members are not to forward/publish what they read outside the list without permission. That's enough to give group #2 the feeling of comfort (depending, always, on how the attitude of the members turns out to be of course) but it doesn't mean it's closed. There are other ways to publish information that might be useful for others, e.g. blogs, wikis, etc. -- +1 to what Guido and Jacob said. I know long time members of this list uncomfortable with the prospect of bringing up anything on this mailing list for various reasons. Given that, and my desire that we provide a safe, friendly forum, biased towards the target audience (newbies) versus those of us who are comfortable with public debate and calcified attitudes, I'm pretty firm the archives stay closed to only members. Anyone, can of course, subscribe. Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Core Mentorship program
On Fri, Mar 25, 2011 at 5:55 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: Guido van Rossum gu...@python.org writes: On Fri, Mar 25, 2011 at 1:57 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: Surely a forum specifically for mentorship will be more useful if outsiders can be directed to existing discussions, without needing to join the private club. This argument comes up repeatedly. Some people object on principle to all closed lists. I apologise if anyone mistakes my position as that. Closed forums are necessary for all kinds of reasons. If my position were against all closed forums, people would be correct to regard that position as too extreme. No, I'm pointing out that a closed forum *for mentorship specifically* is undermining the goal of mentorship: to efficiently share valuable knowledge and help newbies learn from existing discussions with experts and other newbies. One of the great things about a discussion forum open view for the public is that, when a topic comes up again in a *different* forum, I can easily point anyone to the existing discussion without requiring that they join some private group. That's invaluable for spreading knowledge freely. Ben, In principle I agree with you - I would like open archives for the specific reasons you cite, but I value the ability for people who may not be comfortable with coming out and openly discussing things on a list if they know it's open to the magical powers of google and public archives. Heck, having open archives makes it *easier to find out* about the list itself, serving the purpose even more. But - weighed in favor of the target audience (those that may not yet be comfortable with full disclosure, or discussing personality clashes on the tracker, or those worried about future employers digging up stuff) - I want to error on the side of the closed list archives for now. In several months, we all might realize it was a monumental mistake. At that time, we can fix the problem. The perfect is the enemy of the good. :) jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Core Mentorship program
On Mar 25, 2011, at 8:14 PM, Laura Creighton l...@openend.se wrote: In a message of Fri, 25 Mar 2011 18:14:02 -0400, Jesse Noller writes: Ben, In principle I agree with you - I would like open archives for the specific reasons you cite, but I value the ability for people who may not be comfortable with coming out and openly discussing things on a list if they know it's open to the magical powers of google and public archives. Heck, having open archives makes it *easier to find out* about the list itself, serving the purpose even more. But - weighed in favor of the target audience (those that may not yet be comfortable with full disclosure, or discussing personality clashes on the tracker, or those worried about future employers digging up stuff) - I want to error on the side of the closed list archives for now. In several months, we all might realize it was a monumental mistake. At that time, we can fix the problem. I would have thought that the set of people who were more comfortable with the closed list was prettry close to zero. Because the problem with saying something stupid in public is really not one of perfect strangers using google to find out that I said something stupid once, but rather that current members of the target group, in this case the subscribers to python-dev or python-dev-mentors will find out that I think stupid thoughts _now_, and think less of me for it, and maybe say some nasty things about me. Python-dev historically has been rather special. The forbidding message Do not post general Python questions to this list. For help with Python please see the Python help page. in a red boarded box is fairly effective at getting the message do not waste the valuable time of these people across. For a while, I remember, we lied and said that subscriptions to python-dev needed to be approved, even when they didn't. That seemed to deter some people from even trying to join, which was probably either a good thing, or a bad thing on the whole (but, of course, we have no way to measure). And the other thing that makes python-dev unusual is that it is casually read by a large number of people who never say a word. All open mailing lists have lurkers, especially those who read them without a subscription, through some other means, but python-dev is unsual in the number of people who try to read it 'just in case something important happens', and 'just to feel like they know what is going on in the python community'. All of these factors add to the 'don't waste people's time' factor. Thus there is a lot to be said about having a separate group, where python-dev contributors answer questions that they have made time for, even if others might consider them a waste of time. Because those others are not forced to subscribe and read them. But I don't see such a compelling reason for a closed group. It's not as if we expect that mentoring to be a source of deeply personal stories and anecdotes. Or that people want the safety to discuss heretical approaches to changing CPython not expected to go down well with python-dev. It all seems to boil down to 'some people would be more comfortable this way'. I'd like to get some metrics on how many of those people there are. And I'd like to measure them against a different group, people like me who won't contribute to a closed group, in part because the whole closed-ness of it makes me undomfortable. My experience with closed-groups vs open groups has been almost entirely negative, which would be reason enough for me to hesitate to join one, but especially when it comes to a _mentoring_ list. The single most important reason why I would post something I think might be really stupid is because 'if I don't understand this, then there are probably others out there like me in the same boat'. So I ask such things with the hope that the exchange will be googled _a whole lot_ in the future. And again, when I answer a question fully and completely, I do so thinking 'I'll bet a lot of people, and not this one soul, will be interested in this'. If the answer will only be seen by the comparatively few people in a closed mailing list, I am comparatively unmotivated to write anything, or write anything substantial. I've seen a whole lot of very bad behaviour on the part of self-styled leaders of closed mailing lists. They determine the party-line of the group and then, because it is private, blast those souls who do not conform with impunity. Having been on the receiving end of a number of such exchanges, my conclusion has been that having the whole thing open is often the only defence one has. Firstly, most people are more restrained when what they say can be seen by the world at large, so some of these incidents would not happen. But secondly, the ability to share the mail with others greatly empowers the people on the receiving end. But if you cannot get
Re: [Python-Dev] Mentorship list
On Mar 25, 2011, at 8:44 PM, Tommy tommywol...@gmail.com wrote: I was kinda hoping that a private list would have much less noise, and would serve the actual mentoring better. Maybe a mailing list isnt't the ideal tool? That is a hope I would like to see realized. I don't think we will be changing the medium any time soon. So, closed archives it stands for now. Please feel free to join. Jesse___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Core Mentorship program
On Fri, Mar 25, 2011 at 9:59 PM, Glenn Linderman v+pyt...@g.nevcal.com wrote: So... start two mentoring groups, one open, one closed, and see which one survives. I'd rather not. I'd rather walk away from the idea entirely. In fact, this entire thread is quickly becoming an example of why people *don't* want to bring issues to this list. Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GSoC: speed.python.org
Some remarks below. On Mon, Mar 21, 2011 at 2:33 PM, DasIch dasdas...@googlemail.com wrote: Hello Guys, I'm interested in participating in the Google Summer of Code this year and I've been looking at projects in the Wiki, particularly speed.pypy.org[1] as I'm very interested in the current VM development. However given my knowledge that project raised several questions: 1. Up until now the only 3.x Implementation is CPyhon. IronPython, Jython and PyPy don't support it - and to my knowledge won't get support for it during or before GSoC - and could not benefit from it. It seems that a comparison between a Python 2.x implementation and a 3.x implementation is rather pointless; so is this intended to be rather an additional feature to have 3.x there as well? You are correct: But the point of this GSOC project is to do the porting, the initial deployment of speed.python.org will be on the 2.x implementation with 3.x to follow as other VMs migrate. 2. As a follow-up to 1: It is not specified whether the benchmarks should be ported using a tool such as 2to3, if this should not happen or if this is up to the student, this needs clarification. This may be more clear if it were considered under which umbrella this project is actually supposed to happen; will those ported benchmarks end up in CPython or will there be a separate repository for all VMs? We will have a common repository for all benchmarks, for all of the implementations. 3. Several benchmarks (at least the Django and Twisted ones) have dependencies which are not (yet) ported to 3.x and porting those dependencies during GSoC as part of this project is an unrealistic goal. Should those benchmarks, at least for now, be ignored? IMHO: Yes. I think MvL can expand on this as well. [1]: http://wiki.python.org/moin/SpeedDotPythonDotOrg FYI, I recommend coordinating with Miquel Torres (cc'ed) and Maciej Fijalkowski from PyPy on these questions / this project as well. I am currently coordinating getting the hardware setup for this project's hosting. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GSoC: speed.python.org
On Mon, Mar 21, 2011 at 2:48 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 21 Mar 2011 19:33:55 +0100 DasIch dasdas...@googlemail.com wrote: 3. Several benchmarks (at least the Django and Twisted ones) have dependencies which are not (yet) ported to 3.x and porting those dependencies during GSoC as part of this project is an unrealistic goal. Should those benchmarks, at least for now, be ignored? Why not reuse the benchmarks in http://hg.python.org/benchmarks/ ? Many of them are 3.x-compatible. I don't understand why people are working on multiple benchmark suites without cooperating these days. Regards Antoine. Antoine: The goal is to *get* PyPy, Jython, IronPython, etc all using a common set of benchmarks. The idea stems from http://speed.pypy.org/ - those benchmarks grew from the unladen swallow benchmarks, which I think are the ones in http://hg.python.org/benchmarks/. So, yeah - the goal is to get us all reading off the same page. The PyPy people can speak to the changes they made/had to make. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] blogroll for the new blog
On Mon, Mar 21, 2011 at 10:10 PM, Senthil Kumaran orsent...@gmail.com wrote: Doug Hellmann wrote: We are nearly ready to launch the new blog for python-dev. Cool. But I always thought planet.python.org was a kind of blog for python-dev. How will python-dev blog be different? Will add additional redundancy to my RSS reader which gets planet posts as well as individual posts from the same author? This is for developers *of* Python. Doug is just offering to link to your personal account. We are setting up a Development of Python blog which will feature content from CPython, PyPy, etc. Think of it as an announcement location for new/interesting changes. Some of which, yes, might be cross posted to individual authors' blogs depending on if they feel it warrants it. Yes, this question is not related to blogroll but was for my understanding and I am sorry, if it was already discussed elsewhere and I missed it. Perhaps you will be covering this in the introductory post too, if not, it would be a good idea to cover this point. Think of this as a place to say Hey, we just changed the CAPI to be incompatible with 3.2! or PyPy discovered how to bend space time, and is now faster than pure ASM. It's a blog for language development, not general development-with-the language - heck, maybe some of the cooler python-ideas proposals could see the light of day on it. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)
On Sun, Mar 20, 2011 at 7:19 AM, Stefan Behnel stefan...@behnel.de wrote: Nick Coghlan, 12.03.2011 12:43: I posted my rough notes and additional write-ups for Wednesday's VM summit and Thursday's language summit: http://www.boredomandlaziness.org/2011/03/python-vm-summit-rough-notes.html http://www.boredomandlaziness.org/2011/03/python-vm-summit-somewhat-coherent.html http://www.boredomandlaziness.org/2011/03/python-language-summit-rough-notes.html http://www.boredomandlaziness.org/2011/03/python-language-summit-highlights.html It appears that there has been little mention of Cython at the summit, despite of the speed of CPython being a major topic, according to the notes. I can see several areas where Cython could help in speeding up core CPython, one of the most obvious being compilation of standard library modules, and JIT-like compilation of Python modules at import time. Nick mentioned that there was a discussion of making C-only modules available as alternative Python implementations. I'd like to suggest the opposite as well: make pure Python stdlib modules optionally compilable to fast C code using Cython. The obvious advantages are a) speed and b) a single code base for both Python and C modules. Stefan The reason why there was no mention is probably because no one intimately familiar with Cython was there, and if they were - it was not brought up. If Cython supports PyPy - and Jython, and IronPython, your proposal makes sense. The reason for pure python implementation is so that other implementations can share the exact same standard library we have today. I don't see adding a cython version as helping that. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)
On Sun, Mar 20, 2011 at 7:40 AM, Antoine Pitrou solip...@pitrou.net wrote: On Sun, 20 Mar 2011 07:32:34 -0400 Jesse Noller jnol...@gmail.com wrote: The reason why there was no mention is probably because no one intimately familiar with Cython was there, and if they were - it was not brought up. If Cython supports PyPy - and Jython, and IronPython, your proposal makes sense. The reason for pure python implementation is so that other implementations can share the exact same standard library we have today. Well, realistically, they don't. Some functionality just isn't satisfied with a slow Python implementation. The io module is a primary example of that. (but I don't think a Cython version of io would be fast enough, either) Regards Antoine. The intention (and pep that is in progress) is to break out the standard library so that it *is* shared, and used by all of the implementations. We should not go out of our way to make this difficult, and provide pure python implementation whenever we can. This was universally agreed to at the summit at least, and I believe Brett Cannon is going to work on a pep that solidifies it. The other implementations (PyPy, in particular) have run into serious issues with the optimized C versions of modules being placed into the core, and the Python fallbacks being torn out. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)
...snip IMHO, taking modules that currently only have a C implementation due to performance constraints and rewriting them in Cython is a much more worthwhile thing to do than adding an alternative pure Python implementation that other Python runtimes wouldn't use anyway. And at least IronPython could soon benefit directly from a Cython implementation as well. Stefan The other python implementation expressed serious interest, and are willing to help support a shared standard library, and shared modules. So saying they won't use them anyway may apply to things such as the io module, but is far from the truth for the entire standard library. You're also asking that we depend on Cython within core, which while it has it's benefits, I think warrants a PEP and a working implementation to show the potential speedups you're talking about before we can agree to include it/depend on it. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] VM and Language summit info for those not at Pycon (and those that are!)
On Sun, Mar 20, 2011 at 9:39 AM, Stefan Behnel stefan...@behnel.de wrote: Jesse Noller, 20.03.2011 13:51: ...snip IMHO, taking modules that currently only have a C implementation due to performance constraints and rewriting them in Cython is a much more worthwhile thing to do than adding an alternative pure Python implementation that other Python runtimes wouldn't use anyway. And at least IronPython could soon benefit directly from a Cython implementation as well. Stefan The other python implementation expressed serious interest, and are willing to help support a shared standard library, and shared modules. So saying they won't use them anyway may apply to things such as the io module, but is far from the truth for the entire standard library. I guess someone would have to look through the stdlib and make a list of suitable candidates for Cython compilation and/or Python/Cython/C reimplementations at this point. You're also asking that we depend on Cython within core It's a substantial dependency, sure. The question is: what's more work, having to install a specific version of Cython in order to work on CPython, or having to get fluent in C + C-API first? I like the way the Jython people put it, slightly adapted into We write C so you don't have to. which while it has it's benefits, I think warrants a PEP and a working implementation to show the potential speedups you're talking about before we can agree to include it/depend on it. We will have a Cython core developers workshop next weekend. Maybe we can find a bit of time there to run the then-merged-together-bleeding-edge Cython over the standard library and see what that gives. It's not easy to find good benchmarks, though. Most of what the PyPy/Unladen developers settled on so far isn't very interesting for the way Cython works. It's usually a bit of work to make the code substantially faster by providing enough type annotations to let the compiler drop critical Python code into C. We did that for a couple of benchmarks, but lost interest as there wasn't much to win - all we could show was that, by changing the code enough to make it violate the usual constraints of a fair benchmark comparison, you could make it run as fast as C code without writing C code. Not much news to us and nothing that would integrate in any acceptable way into speed.pypy.org. If anyone knows about a good benchmark for a currently pure Python standard library module, preferably a smaller, self-contained one that's somewhat computationally intensive, I'd be happy to hear about it. Stefan The nice thing is about those benchmarks - from PyPy and Unladen Swallow's work - is that they do reflect real world applications, and tend away from pure numerical micro benchmarks (how fast you can do fibonacci is almost worthless). So if we want to add more to that suite of benchmarks, great - but those will be some of the benchmarks against which speed improvements would be measured against. There's also a plan (coming out of the vm summit/language summit) to turn speed.pypy.org into a shared performance.python.org resource, and showcase the speed of CPython, Jython, IronPython and PyPy side by side, using common benchmarks. I am currently working on acquiring the needed resources from the PSF side to get us to the point where the real work can begin. If you can show that Cython lowers the barrier to writing fast(er) Python extensions, without dropping into pure C, and we don't cripple the ability for the other implementation to share the code, then by all means, I personally recommend a PEP. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Please retract my committer rights
On Thu, Mar 17, 2011 at 7:28 PM, Terry Reedy tjre...@udel.edu wrote: On 3/17/2011 1:45 PM, Benjamin Peterson wrote: 2011/3/16 Thomas Hellerthel...@ctypes.org: I would like my committer rights to be retracted. I have been contributing to Python here and there for 10 years now, and it was a pleasant experience. Unfortunately, since about a year I have lots more things to do, and I won't be able to contribute anymore (I have not even started to learn mercurial yet). Plus I'm still using 2.6 or even 2.5 with my own Python projects. Thank you very much for you work. We'd always be happy to have you back. In the meanwhile, it would be nice to have another ctypes maintainer, as there are several open issues. There is certainly a opening for a new person with C experience. I'll ask doug hellmann if this is something we can do a blog post about on the PSF blog. Having another ctypes expert is pretty critical (everyone I know is using it) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] New committers coming online
As agreed to at the language summit, I have pinged the IronPython, Jython and PyPy teams for committers on their respective teams who (do/did not) have commit rights prior to PyCon. These people are: Jeff Hardy (IronPython) Alex Gaynor (PyPy) Carl Friedrich Bolz (PyPy) Maciej Fijalkowski (PyPy) Antonio Cuni (PyPy) And one other whose CLA I have in my bag (still haven't unpacked from PyCon). So if you see them committing, this is a heads up. A side note: The Jython guys would really love to join us in our HG future-land. Frank Wierzbicki expressed a lot of interest in this. Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] public visibility of python-dev decisions before it's too late (was: PyCObject_AsVoidPtr removed from python 3.2 - is this documented?)
On Wed, Mar 9, 2011 at 1:15 AM, Stefan Behnel stefan...@behnel.de wrote: Martin v. Löwis, 08.03.2011 23:47: I think everything here is as it should be. People who really cared about forwards compatibility could have known, but factually, most people don't care enough. Those then learn for the first time that some feature was deprecated after it is actually removed. They then ask why it is removed, and somebody will tell them. I was not aware I could turn on deprecation warning for use of the C API. How can I do that? Not sure that you can. When I said could have known, I meant by reading the documentation. I can confirm that the Cython project was as surprised of the PyCapsule change in Python 3.2 as (I guess) most other users, and I would claim that we are a project with one of the highest probabilities of being impacted by C-API changes. Maybe the what's new document could at least include a link to the relevant python-dev discussion/decision, so that fewer people have to ask back? Actually, why not put up a web page of upcoming changes somewhere, that lists major decisions with user impact that were taken on python-dev? Including a link to the relevant discussion and decision. Often enough, decisions are taken inside of huge mailing list threads that get off-topic before someone has the right idea and everyone who's still there to listen agrees. Even for people lurking around on python-dev, it's easy enough to miss these moments. A publicly visible list of those decisions would a) make it easier for non-core developers to follow important changes on python-dev b) allow potentially impacted people to bring up their POV more quickly, e.g. during the alpha cycle of a deprecation release rather than the following release, as in this case c) document the decision taking process by listing the relevant mailing list threads d) help in writing the what's new documents Stefan A python dev blog/news site is on the list of topics for the language summit tomorrow at PyCon. Doug hellmann is going to be managing that discussion - but there is a general agreement we have to spread word outside of core and into the general world a little bit more - and this also applies to other python VMs as well. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] CPython hg transition complete
On Sat, Mar 5, 2011 at 12:39 PM, Georg Brandl g.bra...@gmx.net wrote: I'm very happy to announce that the core Python repository switch to Mercurial is complete and the new repository at http://hg.python.org/cpython/ is now officially open for cloning, and for commits by those who had commit access to SVN. The developers' guide at http://docs.python.org/devguide/ has been updated to talk about Mercurial and should be enough to get anyone started with a clone. We'll work on extracting active feature branches into separate clones next; please let us know which branches these are (we already know of py3k-cdecimal, pep-3151 and pep-382). To make new feature branches (ie. clones) that are to be available at hg.python.org, best use the server side clone feature that is available at http://hg.python.org/cpython/ in order to create the new repository. To look up SVN revisions, use hg.python.org/lookup/rX. The tracker has also been updated to link to hg.python.org for files and revisions. For the future, it will recognize hg changeset hashes as well (without brackets, see the recent thread). The buildbots should also now be building from the hg repositories. Please let me know if you notice any disruptions, or anything else that needs fixing, or any other question. Also please redirect praise to Antoine Pitrou and Dirkjan Ochtman who did most of the actual work. Georg Thanks to all of you for doing this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] contributors survey?
On Wed, Mar 2, 2011 at 7:07 AM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 1 Mar 2011 20:43:27 -0800 Guido van Rossum gu...@python.org wrote: But I wouldn't be surprised if some people had regrets about the way the community works (I can recall at least one such case) and it would be useful to learn from those occasions, if they'll let us. And the numbers might tell us something, too. Yes, that's the kind of things that would be good to hear about IMO. It's obvious that in some cases patches and reports go simply unanswered for years, and in these cases a first-time reporter or contributor won't bother again (who would?). But I wonder if there are other social or technical factors, such as the community being too intimidating or not welcoming enough. Actually, if some python-dev readers have something to say about that, they are welcome :) FWIW, Here's some feedback I got from the community awhile ago - not all of the respondents are ex contributors, but rather this is a general why don't you contribute question. I've still not had the time to internalize it, other then to pester Brett to work on the dev docs. http://jessenoller.com/2010/04/22/why-arent-you-contributing-to-python/ http://news.ycombinator.com/item?id=1285897 http://www.reddit.com/r/Python/comments/burio/why_arent_you_contributing_to_python/ It's worth a good read-through. I got a lot of private emails all in the same tone. Speed of turn around, push back from entrenched developers turning off new contributors, etc. Jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] contributors survey?
On Wed, Mar 2, 2011 at 10:00 AM, Jesse Noller jnol...@gmail.com wrote: On Wed, Mar 2, 2011 at 7:07 AM, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 1 Mar 2011 20:43:27 -0800 Guido van Rossum gu...@python.org wrote: But I wouldn't be surprised if some people had regrets about the way the community works (I can recall at least one such case) and it would be useful to learn from those occasions, if they'll let us. And the numbers might tell us something, too. Yes, that's the kind of things that would be good to hear about IMO. It's obvious that in some cases patches and reports go simply unanswered for years, and in these cases a first-time reporter or contributor won't bother again (who would?). But I wonder if there are other social or technical factors, such as the community being too intimidating or not welcoming enough. Actually, if some python-dev readers have something to say about that, they are welcome :) FWIW, Here's some feedback I got from the community awhile ago - not all of the respondents are ex contributors, but rather this is a general why don't you contribute question. I've still not had the time to internalize it, other then to pester Brett to work on the dev docs. http://jessenoller.com/2010/04/22/why-arent-you-contributing-to-python/ http://news.ycombinator.com/item?id=1285897 http://www.reddit.com/r/Python/comments/burio/why_arent_you_contributing_to_python/ It's worth a good read-through. I got a lot of private emails all in the same tone. Speed of turn around, push back from entrenched developers turning off new contributors, etc. Jesse Let me point out, in a positive light, that the feedback from the above is what triggered me to drive the PSF Sprints project (http://pythonsprints.com/) at the board/PSF level. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] contributors survey?
On Tue, Mar 1, 2011 at 4:24 PM, Antoine Pitrou solip...@pitrou.net wrote: Hello, In http://mail.python.org/pipermail/python-committers/2011-February/001340.html, I was asking whether it would be useful to make a survey of past contributors, as in: First, we did a survey of all our past developers who had left the project, asking them why they had left. This was just a free-form survey, allowing people to answer any way they wanted. (from the quoted article in the thread linked above) Since I didn't get any answer, I wonder if the idea simply got overlooked, or if there's no need at all. Regards Antoine. I think doing a survey like this is a *really* good idea. We have an account at our disposal (as the PSF) at http://www.surveymonkey.com/ which we use for PyCon feedback. Maybe we can leverage that, if we can come up with good questions? jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] contributors survey?
On Tue, Mar 1, 2011 at 9:22 PM, Stephen J. Turnbull step...@xemacs.org wrote: Antoine Pitrou writes: Following the example given in the original article, I was considering a single freeform question: why did you stop contributing after your last patch to CPython? (of course, that text should be decorated with a greeting and an introduction and the wording can be improved). Does anybody ever stop contributing? Occasionally, yes, but in most cases it's just that this interval between explicit contributions (usually patches, but also reviews, PEPs, mailing list posting, bug reports, ...) is longer than the period since the last one. :-) How about Hello. We [the PSF?] would like to thank you for your past patches to CPython, and take this opportunity to learn something about how to improve our workflow. We would appreciate your cooperation in answering the following question. It has been more than one year since your last patch to CPython. We would like to understand why it's been so long [, and if there is anything we could do to help you contribute patches more frequently]. The clause in brackets is outside the scope of Antoine's wording, but I assume that's where we're going with this. Personally, I like this, but I would skip the PSF aspect of it, and focus on core. The PSF exists outside of the domain of CPython and should probably avoid taking too much credit or inserting itself more. :) jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's get PEP 380 into Python 3.3
On Fri, Feb 25, 2011 at 5:43 PM, Guido van Rossum gu...@python.org wrote: Now that the language moratorium is lifted, let's make sure to get PEP 380 implemented for Python 3.3. I think there are some minor issues to be resolved, but I don't think that should stop someone from doing a first pass of the implementation (especially since a version for 2.6 already exists). (OTOH I am not much enamored with cofunctions, PEP 3152.) Yay! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 3.2
On Wed, Feb 23, 2011 at 5:45 PM, Barry Warsaw ba...@python.org wrote: On Feb 21, 2011, at 12:39 AM, Victor Stinner wrote: Le dimanche 20 février 2011 à 23:22 +0100, Georg Brandl a écrit : On behalf of the Python development team, I'm delighted to announce Python 3.2 final release. Python 3.2 is a continuation of the efforts to improve and stabilize the Python 3.x line. Congratulation to all Python developers for this wonderful release! And a special kudo to our release manager, Georg. Indeed, great job Georg. I hereby nominate you for Python 3.3 RM. No good deed goes unpunished. :) I hope that Python 3 is now stable enough to support migration of major projects like Django, Twisted or Zope. Other important projets like Distribute, Jinja2, PyQt, PyGObject, pygame, NumPy+SciPy and Sphinx are already compatible with Python 3. Agreed! I hope porting to Python 3 can be a major theme for Pycon this year. -Barry It is. Trust me. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 10:28 AM, Martin v. Löwis mar...@v.loewis.de wrote: I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped. Regards, Martin The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 8:17 AM, Victor Stinner victor.stin...@haypocalc.com wrote: Hi, FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of a concurrent.futures failure. The problem is that test_concurrent_futures uses many (multiprocessing) POSIX semaphores, whereas POSIX semaphores support in FreeBSD is recent and limited. We have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: * http://bugs.python.org/issue10348 * Too many open files errors on x86 FreeBSD 7.2 3.x buildbot ^-- thread in python-dev opened last month I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to write a patch, but I never used semaphores (POSIX or SysV). There is a third party module which looks complete and stable: http://semanchuk.com/philip/sysv_ipc/ It is released under the BSD license. It supports semaphores, but also shared memory and message queues. We don't need all of those, semaphores would be enough. I added its author (Philip Semanchuk) to this thread. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Victor The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot. Moving from POSIX IPC to SYSV should *not* be on the plate for a release blocker - that's a much larger task. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Wed, Dec 29, 2010 at 1:34 PM, Martin v. Löwis mar...@v.loewis.de wrote: Am 29.12.2010 18:54, schrieb Jesse Noller: On Wed, Dec 29, 2010 at 10:28 AM, Martin v. Löwis mar...@v.loewis.de wrote: I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped. Regards, Martin The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. I'm not so sure that skipping the test is the right approach. Doesn't that mean that the code will still fail at runtime with difficult-to-explain messages? I'd rather prefer if the functionality wasn't available in the first place. Also, what specific test are you referring to? Regards, Martin If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 3:49 PM, Martin v. Löwis mar...@v.loewis.de wrote: If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Dec 29, 2010, at 4:54 PM, Martin v. Löwis mar...@v.loewis.de wrote: Am 29.12.2010 22:34, schrieb Jesse Noller: On Dec 29, 2010, at 3:49 PM, Martin v. Löwis mar...@v.loewis.de wrote: If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. So what do you propose to do? I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. I would document the limitation in the futures/multiprocessing documentation, and skip that particular test for now on FreeBSD. FreeBSDs limitations seem arbitrary, but I'm not a FBSD expert. This isn't a bug in either futures or multiprocessing though. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, Nov 3, 2010 at 3:45 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 03 Nov 2010 19:26:53 + Michael Foord fuzzy...@voidspace.org.uk wrote: Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature... Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of pickling a TestCase could be, and what the use cases are. Regards Antoine. Splitting groups of tests to run in parallel via multiple processes is a pretty good use case. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multiprocessing maintenance
On Sat, Oct 23, 2010 at 2:20 PM, Antoine Pitrou solip...@pitrou.net wrote: You mean: actively feeling responsible for it? I guess nobody - as for many other modules in the standard library. Or do you mean: who is willing to work on it, in principle? Both. Originally the module is/was meant to be officially maintained by Jesse, as far as I understand. But bugs filed against multiprocessing have been lingering in the tracker for quite a long time. Regards Antoine. Like Ask said, we both are, and still feel responsible. Finding time is another problem altogether, and it's been in short supply. In short: No, neither one of us has walked away. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multiprocessing maintenance
On Sat, Oct 23, 2010 at 2:10 PM, Martin v. Löwis mar...@v.loewis.de wrote: Who is doing multiprocessing maintenance these days? I thought Ask Solem had been given commit privs for that, but I haven't seen any activity from him; and Jesse is, mostly, absent. Is anyone working on the multiprocessing issues? (no, I'm not planning to address them :-)) You mean: actively feeling responsible for it? I guess nobody - as for many other modules in the standard library. Or do you mean: who is willing to work on it, in principle? The last committers are georg.brandl, gregory.p.smith, martin.v.loewis, and antoine.pitrou. Both Ask and myself feel actively responsible. Antoine could have just as easily asked privately. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug week-end on the 20th-21st?
On Sat, Oct 23, 2010 at 1:08 PM, Antoine Pitrou solip...@pitrou.net wrote: Hello, The first 3.2 beta is scheduled by Georg for November 13th. What would you think of scheduling a bug week-end one week later, that is on November 20th and 21st? We would need enough core developers to be available on #python-dev. Regards Antoine. If anyone wants to spin up local, in person presences for this - namely, small local sprints, let me / the sprints team know. We (The PSF) have money to help fund. This would be the perfect use of the resources - more info http://pythonsprints.com/ jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multiprocessing maintenance
On Mon, Oct 25, 2010 at 7:19 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 25 Oct 2010 10:01:43 -0400 Jesse Noller jnol...@gmail.com wrote: On Sat, Oct 23, 2010 at 2:10 PM, Martin v. Löwis mar...@v.loewis.de wrote: Who is doing multiprocessing maintenance these days? I thought Ask Solem had been given commit privs for that, but I haven't seen any activity from him; and Jesse is, mostly, absent. Is anyone working on the multiprocessing issues? (no, I'm not planning to address them :-)) You mean: actively feeling responsible for it? I guess nobody - as for many other modules in the standard library. Or do you mean: who is willing to work on it, in principle? The last committers are georg.brandl, gregory.p.smith, martin.v.loewis, and antoine.pitrou. Both Ask and myself feel actively responsible. Antoine could have just as easily asked privately. Well, the question and its eventual answer were meant for public consumption, not for my personal edification. Thanks for answering anyway. We can then continue to put you and Ask in the nosy lists for multiprocessing issues. Regards Antoine. I'm not pleased with the current state of affairs and I am working to change this. I apologize for the delay on the bugs. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyCon 2011 Reminder: Call for Proposals, Posters and Tutorials - us.pycon.org
PyCon 2011 Reminder: Call for Proposals, Posters and Tutorials - us.pycon.org === Well, it's October 25th! The leaves have turned and the deadline for submitting main-conference talk proposals expires in 7 days (November 1st, 2010)! We are currently accepting main conference talk proposals: http://us.pycon.org/2011/speaker/proposals/ Tutorial Proposals: http://us.pycon.org/2011/speaker/proposals/tutorials/ Poster Proposals: http://us.pycon.org/2011/speaker/posters/cfp/ PyCon 2011 will be held March 9th through the 17th, 2011 in Atlanta, Georgia. (Home of some of the best southern food you can possibly find on Earth!) The PyCon conference days will be March 11-13, preceded by two tutorial days (March 9-10), and followed by four days of development sprints (March 14-17). We are also proud to announce that we have booked our first Keynote speaker - Hilary Mason, her bio: Hilary is the lead scientist at bit.ly, where she is finding sense in vast data sets. She is a former computer science professor with a background in machine learning and data mining, has published numerous academic papers, and regularly releases code on her personal site, http://www.hilarymason.com/. She has discovered two new species, loves to bake cookies, and asks way too many questions. We're really looking forward to having her this year as a keynote speaker! Remember, we've also added an Extreme talk track this year - no introduction, no fluff - only the pure technical meat! For more information on Extreme Talks see: http://us.pycon.org/2011/speaker/extreme/ We look forward to seeing you in Atlanta! Please also note - registration for PyCon 2011 will also be capped at a maximum of 1,500 delegates, including speakers. When registration opens (soon), you're going to want to make sure you register early! Speakers with accepted talks will have a guaranteed slot. We have published all registration prices online at: http://us.pycon.org/2011/tickets/ Important Dates November 1st, 2010: Talk proposals due. December 15th, 2010: Acceptance emails sent. January 19th, 2011: Early bird registration closes. March 9-10th, 2011: Tutorial days at PyCon. March 11-13th, 2011: PyCon main conference. March 14-17th, 2011: PyCon sprints days. Contact Emails: Van Lindberg (Conference Chair) - v...@python.org Jesse Noller (Co-Chair) - jnol...@python.org PyCon Organizers list: pycon-organiz...@python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] We should be using a tool for code reviews
On Wed, Sep 29, 2010 at 2:32 PM, Guido van Rossum gu...@python.org wrote: I would like to recommend that the Python core developers start using a code review tool such as Rietveld or Reviewboard. I don't really care which tool we use (I'm sure there are plenty of pros and cons to each) but I do think we should get out of the stone age and start using a tool for the majority of our code reviews. While I would personally love to see Rietveld declared the official core Python code review tool, I realize that since I wrote as a Google engineer and it is running on Google infrastructure (App Engine), I can't be fully objective about the tool choice -- even though it is open source, has several non-Googler maintainers, and can be run outside App Engine as well. But I do think that using a specialized code review tool rather than unstructured email plus a general-purpose issue tracker can hugely improve developer performance and also increase community participation. (A code review tool makes it much more convenient for a senior reviewer to impart their wisdom to a junior developer without appearing judgmental or overbearing.) See also this buzz thread: http://www.google.com/buzz/115212051037621986145/At6Rj82Kret/When-will-the-Python-dev-community-start-using -- --Guido van Rossum (python.org/~guido) Regardless of the tool(s) used, code reviews are a fantastic equalizer. If you have long time, experienced developers submitting to the same rules that newer contributors have to follow then it helps remove the idea that there is special treatment occurring. Additionally, a lot of people are terrified of code reviews as they view it as a public flogging - holding everyone to the same standards, and showing this is not the case helps fight this perception. Not to mention; there's a lot to be learned from doing them on both sides. At work, I learn about chunks of code I might not have otherwise known about or approaches to a problem I'd never considered. I sort of drank the kool-aid. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] We should be using a tool for code reviews
On Thu, Sep 30, 2010 at 10:52 AM, exar...@twistedmatrix.com wrote: On 02:47 pm, jnol...@gmail.com wrote: On Wed, Sep 29, 2010 at 2:32 PM, Guido van Rossum gu...@python.org wrote: I would like to recommend that the Python core developers start using a code review tool such as Rietveld or Reviewboard. I don't really care which tool we use (I'm sure there are plenty of pros and cons to each) but I do think we should get out of the stone age and start using a tool for the majority of our code reviews. While I would personally love to see Rietveld declared the official core Python code review tool, I realize that since I wrote as a Google engineer and it is running on Google infrastructure (App Engine), I can't be fully objective about the tool choice -- even though it is open source, has several non-Googler maintainers, and can be run outside App Engine as well. But I do think that using a specialized code review tool rather than unstructured email plus a general-purpose issue tracker can hugely improve developer performance and also increase community participation. (A code review tool makes it much more convenient for a senior reviewer to impart their wisdom to a junior developer without appearing judgmental or overbearing.) See also this buzz thread: http://www.google.com/buzz/115212051037621986145/At6Rj82Kret/When- will-the-Python-dev-community-start-using -- --Guido van Rossum (python.org/~guido) Regardless of the tool(s) used, code reviews are a fantastic equalizer. If you have long time, experienced developers submitting to the same rules that newer contributors have to follow then it helps remove the idea that there is special treatment occurring. Of course, this is only true if the core developers *do* submit to the same rules. Is anyone proposing that current core committers have all their work reviewed before it is accepted? (I am strongly in favor of this, but I don't think many core committers are.) Jean-Paul I'll propose it, knowing full well I won't win. Code reviews have saved my bacon on numerous occasions. The best unit tests on the planet won't protect you against a fundamentally bad assumption or logic error. Like I said - I think it helps equalize things. YMMV. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] We should be using a tool for code reviews
On Thu, Sep 30, 2010 at 12:53 PM, geremy condra debat...@gmail.com wrote: On Thu, Sep 30, 2010 at 9:33 AM, Barry Warsaw ba...@python.org wrote: On Sep 30, 2010, at 10:47 AM, Jesse Noller wrote: Not to mention; there's a lot to be learned from doing them on both sides. At work, I learn about chunks of code I might not have otherwise known about or approaches to a problem I'd never considered. I sort of drank the kool-aid. Tools aside, I completely agree. Many projects that I contribute to have policies such as nothing lands without reviewer approval. For some that means one reviewer must approve it, for others two +1s and no -1s, or a coding approval and a ui approval, etc. When I was the review team lead on Launchpad, I had a goal that every developer would also eventually be a reviewer. We started with a small number of experienced developers, then ran a mentor program to train new reviewers (who we called mentats). A mentat approval was not enough to land a branch, but the mentor could basically say yes, i agree with the review and it would land. Eventually, by mutual consent a mentat would graduate to full reviewership (and hopefully be a mentor to new reviewers). This was hugely successful among many dimensions. It got everyone on the same page as to coding standards, it equalized the playing field, it got everyone to think collaboratively as a team, folks learned about parts of the system they didn't have day-to-day intimate knowledge about, and it got changes landed much more quickly. Here are a few things we learned along the way. Their applicability to Python will vary of course, and we need to find what works for us. * Keep branches *small*. We had a limit of 800 lines of diff, with special explicit permission from the person reviewing your change to exceed. 800 lines is about the maximum that a person can review in a reasonable amount of time without losing focus. * The *goal* was 5 minutes review, but the reality was that most reviews took about 15-20 minutes. If it's going longer, you weren't doing it right. This meant that there was a level of trust between the reviewer and the dev, so that the basic design had been previously discussed and agreed upon (we mandated pre-implementation chats), etc. A reviewer was there to sanity check the implementation, watch for obvious mistakes, ensure test coverage for the new code, etc. If you were questioning the basic design, you weren't doing a code review. It was okay to reject a change quickly if you found fatal problems. * The primary purpose of a code review was to learn and teach, and in a sense, the measurable increase in quality was a side-effect. What I mean by that is that the review process cannot catch all bugs. It can reduce them, but it's much more valuable to share expertise on how to do things. E.g. Did you know that if X happens, you won't be decref'ing Y? We like to use goto statements to ensure that all objects are properly refcounted even in the case of exceptional conditions. That's a teaching moment that happens to improve quality. * Reviews are conversations, and it's a two way street. Many times a dev pushed back on one of my suggestions, and we'd have weekly reviewer meetings to hash out coding standards differences. E.g. Do you check for empty sequences with if len(foo) == 0 or if not foo? The team would make those decisions and you'd live by them even if you didn't agree. It's also critical to document those decisions, and a wiki page style guide works very well (especially when you can point to PEP 8 as your basic style guide for example). * Reviews are collaborations. You're both there to get the code landed so work together, not at odds. Try to reach consensus, and don't be afraid to compromise. All code is ultimately owned by everyone and you never know who will have to read it 2 years from now, so keep things simple, clear, and well commented. These are all aesthetics that a reviewer can help with. * As a reviewer ASK QUESTIONS. The best reviews were the ones that asked lots of questions, such as have you thought about the race conditions this might introduce? or what happens when foo is None? A reviewer doesn't necessarily have to guess or work out every detail. If you don't understand something, ask a question and move on. Let the coder answer it to your satisfaction. As a reviewer, once all your questions are answered, you know you can approve the change. * Keep reviews fast, easy, and fun. I think this is especially true for Python, where we're all volunteers. Keeping it fast, easy, and fun greatly improves the odds that code will be reviewed for the good of the project. * Have a dispute resolution process. If a reviewer and coder can't agree, appeal to a higher authority. As review team leader, I did occasionally have to break ties. * Record
Re: [Python-Dev] Mark PEP 3148 as Final?
On Mon, Sep 27, 2010 at 5:09 PM, Nick Coghlan ncogh...@gmail.com wrote: I saw the code for PEP 3148 go by on python-checkins the other day. Is there anything left to be done on that front, or can the PEP be marked Final? Cheers, Nick. Argh, yes :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] WSGI is now Python 3-friendly
On Sat, Sep 25, 2010 at 3:56 PM, P.J. Eby p...@telecommunity.com wrote: I have only done the Python 3-specific changes at this point; the diff is here if anybody wants to review, nitpick or otherwise comment: http://svn.python.org/view/peps/trunk/pep-0333.txt?r1=85014r2=85013pathrev=85014 For that matter, if anybody wants to take a crack at updating Python 3's wsgiref based on the above, feel free. ;-) I'll be happy to answer any questions I can that come up in the process. (Please note: I went with Ian Bicking's headers are strings, bodies are bytes proposal, rather than my original bodies and outputs are bytes one, as there were not only some good arguments in its favor, but because it also resulted in fewer changes to the PEP, especially in the code samples.) I will continue to work on adding the other addenda/errata mentioned here: http://mail.python.org/pipermail/web-sig/2010-September/004655.html But because these are shoulds rather than musts, and apply to both Python 2 and 3, they are not as high priority for immediate implementation in wsgiref and do not necessarily need to hold up the 3.2 release. (Nonetheless, if anybody is willing to implement them in the Python 3 version, I will happily review the changes for backport into the Python 2 standalone version of wsgiref, and issue an updated release to include them.) Thanks! This looks like good progress (and does not seem to block the progress of PEP 444) but would it be possible to do it as a different PEP rather then just an update; or adding an explicit these are the differences between v1 and v2 section? It seems like it will end up different enough to be a different specification, closely related to the original, but different enough to trip up all the people maintaining current WSGI servers and apps. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving the developer docs?
On Thu, Sep 23, 2010 at 6:11 AM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 23 Sep 2010 00:29:51 -0400 Fred Drake fdr...@acm.org wrote: On Wed, Sep 22, 2010 at 10:38 PM, Brett Cannon br...@python.org wrote: the first thing on the agenda is a complete rewrite of the developer docs and moving them into the Doc/ directory I'd like to know why you think moving the developer docs into the CPython tree makes sense. My own thought here is that they're not specific to the version of Python, though some of the documentation deals with the group of specific branches being maintained. Many parts of the library docs aren't version-specific either :) The dev docs may differ slightly from one version to another, for example if a version introduces some new possibilities for tooling, or far-reaching implementation changes (think Unladen Swallow). The practicality argument of being able to edit those docs without having to master a separate (pydotorg) workflow sounds quite strong to me. Agreed with Antoine here the additional workflow/repo/build process/etc sucks Besides - who cares if only a subset of users would be interested in our workflow? If it's more than 0, and it helps bring on new contributors, who cares? If we can make it easier to maintain information, and find that information, why not do it? jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving the developer docs?
On Thu, Sep 23, 2010 at 10:01 AM, Barry Warsaw ba...@python.org wrote: On Sep 23, 2010, at 08:40 AM, Georg Brandl wrote: That's right. It is true that it isn't branch-specific information, and that does cause a little bit of irritation for me too, but neither is Misc/developers.txt or Misc/maintainers.rst. Of course, we might consider a separate HG repository (I'm all in favor of many small repos, instead of a gigantic sandbox one). The downside is that I really like the developer docs at docs.python.org, and it would complicate the build process a bit. Ideally, I would really like to see the developer docs live outside the CPython source repository. There's no reason to tie the dev docs to CPython's svn merge policies, write acls, or release schedules. Given the way docs.python.org is stitched together, and the fact that we (still ;) haven't moved to a dvcs, this may not be feasible. These docs are better off in the wiki than in the source tree. -Barry -1 on wiki; wikis are where good information goes off to die. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving the developer docs?
On Thu, Sep 23, 2010 at 10:35 AM, Barry Warsaw ba...@python.org wrote: On Sep 23, 2010, at 10:06 AM, Jesse Noller wrote: -1 on wiki; wikis are where good information goes off to die. Well, *all* documentation requires vigilance to remain relevant and current. I'm sure you don't think the Python wiki is useless, right? ;) I do. I visit it as little as possible. :( ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
