M.-A. Lemburg wrote:
Backporting PyCapsule is fine, but the changes you made to all
those PyCObject uses does not look backwards compatible.

The C APIs exposed by the modules (e.g. the datetime module)
are used in lots of 3rd party extension modules and changing
them from PyCObject to PyCapsule is a major change in the
module API.

You're right, my changes aren't backwards compatible. I thought it was reasonable for four reasons:

1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a few lines by mixing and matching CObjects. Switching Python to capsules prevents a class of exploits. I've included a script at the bottom of this message that demonstrates three such crashes. The script runs in Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

2. As I just mentioned, Python 3.1 already uses capsules everywhere instead of CObjects. Since part of the purpose of Python 2.7 is to prepare developers for the to upgrade to 3.1, getting them to switch to capsules now is just one more way they are prepared.

3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we ever made a 2.8 I want to remove it completely.

4. When Python publishes an API using a CObject, it describes the thing the CObject points to in a header file. In nearly all cases that header file also provides a macro or inline function that does the importing work for you. I changed those to use capsules too. So if the third-party code uses the macro or inline function, all you need do is recompile it against 2.7 and it works fine. Sadly I know of one exception: pyexpat.expat_CAPI. The header file just describes the struct pointed to by the CObject, but callers


I can suggest four ways to ameliorate the problem.

First, we could do as Antoine Pitrou suggests on the bug (issue 7992): wherever the CObject used to be published as a module attribute to expose an API, we could provide both a CObject and a capsule; internally Python would only use the capsules. This would allow third-party libraries to run against 2.7 unchanged. The major problem with this is that third-party libraries would still be vulnerable to the mix-and-match CObject crash. A secondary, minor concern: obviously we'd store the CObject attribute with the existing name, and the capsule attribute would have to get some new name. But in Python 3.1, these attributes already expose a capsule. Therefore, people who convert to using the capsules now would have to convert again when moving to 3.1.

Second, we could make CObject internally support unpacking capsules. If you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and return the pointer within. (We could probably also map the capsule "context" to the CObject "desc", if any of the Python use cases needed it.) I wouldn't change anything else about CObjects; creating and using them would continue to work as normal. This would also allow third-party libraries to run against Python 2.7 unchanged. The only problem is that it's unsafe, as indeed allowing any use of PyCObject_AsVoidPtr() is unsafe.

Third, I've been pondering writing a set of preprocessor macros, shipped in their own header file distributed independently of Python and released to the public domain, that would make it easy to use either CObjects or capsules depending on what version of Python you were compiling against. Obviously, using these macros would require a source code change in the third-party library. But these macros would make it a five-minute change. This could compliment the first or second approaches.

Fourth, we could back out of the changes to published APIs and convert them back to CObjects. -1.


Your thoughts?


/larry/

-----

import sys
def log(message):
   print(message)
   sys.stdout.flush()

def crash1():
   log("Running crash1...")
   try:
       import datetime
       import cStringIO
       cStringIO.cStringIO_CAPI = datetime.datetime_CAPI

       import cPickle
       s = cPickle.dumps([1, 2, 3])
   except ImportError:
       # This test isn't translatable to Python 3.
       pass
   log("Survived crash1!")


def crash2():
   log("Running crash2...")
   try:
       import unicodedata
       import _socket
       _socket.CAPI = unicodedata.ucnhash_CAPI
       import ssl
   except AttributeError:
       # Congratulations, you didn't crash.
       pass
   log("Survived crash2!")


def crash3():
   log("Running crash3...")
   try:
       import unicodedata
       import _multibytecodec
       _multibytecodec.__create_codec(unicodedata.ucnhash_CAPI)

   except ValueError:
       # Congratulations, you didn't crash.
       pass
   log("Survived crash3!")

import sys

if len(sys.argv) > 1:
   if sys.argv[1] == '1':
       crash1()
       sys.exit(0)
   elif sys.argv[1] == '2':
       crash2()
       sys.exit(0)
   elif sys.argv[1] == '3':
       crash3()
       sys.exit(0)

crash1()
crash2()
crash3()
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Reply via email to