Re: [Python-Dev] Py_BuildValue and decref
On Sun, Sep 10, 2006 at 07:35:59PM +1200, Greg Ewing wrote: Barry Warsaw wrote: I just want to point out that the C API documentation is pretty silent about the refcounting side-effects in error conditions (and often in success conditions too) of most Python functions. For example, what is the refcounting side-effects of PyDict_SetItem() on val? What about if that function fails? Has val been incref'd or not? What about the side-effects on any value the new one replaces, both in success and failure? The usual principle is that the refcounting behaviour is (or should be) independent of whether the function succeeded or failed. In the absence of any statement to the contrary in the docs, you should be able to assume that. The words used to describe the refcount behaviour of some functions can be rather confusing, but it always boils down to one of two cases: either the function borrows a reference (and does its own incref if needed, the caller doesn't need to care) or it steals a reference (so the caller is always responsible for doing an incref if needed before calling). What that rather convoluted comment about PyTuple_SetItem is trying to say is just that it *always* steals a reference, regardless of whether it succeeds or fails. I expect the same is true of Py_BuildValue. Given that it doesn't seem to be the case, and my quick look at the code indicates that even internally python is inconsistent, should I file a low-severity bug so we don't lose track of this? Misa ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Py_BuildValue and decref
Hi,
Looking at:
http://docs.python.org/api/arg-parsing.html
The description for O is:
O (object) [PyObject *]
Store a Python object (without any conversion) in a C object pointer. The
C program thus receives the actual object that was passed. The object's
reference count is not increased. The pointer stored is not NULL.
There is no description of what happens when Py_BuildValue fails. Will it
decref the python object passed in? Will it not?
Looking at tupleobject.h:
/*
Another generally useful object type is a tuple of object pointers.
For Python, this is an immutable type. C code can change the tuple items
(but not their number), and even use tuples are general-purpose arrays of
object references, but in general only brand new tuples should be mutated,
not ones that might already have been exposed to Python code.
*** WARNING *** PyTuple_SetItem does not increment the new item's reference
count, but does decrement the reference count of the item it replaces,
if not nil. It does *decrement* the reference count if it is *not*
^^^
inserted in the tuple. Similarly, PyTuple_GetItem does not increment the
returned item's reference count.
*/
So, if the call to PyTuple_SetItem fails, the value passed in is lost. Should
I expect the same thing with Py_BuildValue?
Looking at how other modules deal with this, I picked typeobject.c:
result = Py_BuildValue([O], (PyObject *)type);
if (result == NULL) {
Py_DECREF(to_merge);
return NULL;
}
so no attempt to DECREF type in the error case.
Further down...
if (n) {
state = Py_BuildValue((NO), state, slots);
if (state == NULL)
goto end;
}
and further down:
end:
Py_XDECREF(cls);
Py_XDECREF(args);
Py_XDECREF(args2);
Py_XDECREF(slots);
Py_XDECREF(state);
Py_XDECREF(names);
Py_XDECREF(listitems);
Py_XDECREF(dictitems);
Py_XDECREF(copy_reg);
Py_XDECREF(newobj);
return res;
so it will attempt to DECREF the (non-NULL) slots in the error case.
It's probably not a big issue since if Py_BuildValue fails, you have bigger
issues than memory leaks, but it seems inconsistent to me. Can someone that
knows the internal implementation clarify one way over the other?
Thanks!
Misa
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_BuildValue and decref
On Fri, Sep 08, 2006 at 06:27:08PM -0400, Barry Warsaw wrote:
On Sep 8, 2006, at 6:06 PM, Mihai Ibanescu wrote:
There is no description of what happens when Py_BuildValue fails.
Will it
decref the python object passed in? Will it not?
I just want to point out that the C API documentation is pretty
silent about the refcounting side-effects in error conditions (and
often in success conditions too) of most Python functions. For
example, what is the refcounting side-effects of PyDict_SetItem() on
val? What about if that function fails? Has val been incref'd or
not? What about the side-effects on any value the new one replaces,
both in success and failure?
In this particular case, it doesn't decref it (or so I read the code).
Relevant code is in do_mkvalue from Python/modsupport.c
case 'N':
case 'S':
case 'O':
if (**p_format == '') {
typedef PyObject *(*converter)(void *);
converter func = va_arg(*p_va, converter);
void *arg = va_arg(*p_va, void *);
++*p_format;
return (*func)(arg);
}
else {
PyObject *v;
v = va_arg(*p_va, PyObject *);
if (v != NULL) {
if (*(*p_format - 1) != 'N')
Py_INCREF(v);
}
else if (!PyErr_Occurred())
/* If a NULL was passed
* because a call that should
* have constructed a value
* failed, that's OK, and we
* pass the error on; but if
* no error occurred it's not
* clear that the caller knew
* what she was doing. */
PyErr_SetString(PyExc_SystemError,
NULL object passed to
Py_BuildValue);
return v;
}
Barry, where can I ship you my cloning machine? :-)
Misa
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] logging module broken because of locale
On Thu, Jul 20, 2006 at 05:09:38AM -0500, [EMAIL PROTECTED] wrote: Misa Good point. Does the attached patch look reasonable? ... Misa -self.when = string.upper(when) Misa +self.when = unicode(when).upper() ... The use of the string module instead of string methods suggests to me that the logging package attempts to work with older versions of Python. Looking at PEP 291 it looks like 1.5.2 compatibility is desired (no string methods, no unicode). I think a conscious decision by someone (probably Vinay Sajip) to give up that compatibility would be required. Agreed. There is a note that the module should work with python = 1.5.2 at the top of the module. It's up to Vinay to decide if we want to drop support for 1.5.2 in the module included in newer pythons, or the attached patch would make it work for 1.5.2 as well (as in it's not more broken than before). I would like to redo the patch once more to get rid of the try-except and use __builtins__ instead (but for some reason it kept jumping from being a module to being a dictionary and I just wanted the proof of concept). Misa --- Python-2.4.3/Lib/logging/handlers.py.nolocale 2006-07-19 12:15:46.0 -0400 +++ Python-2.4.3/Lib/logging/handlers.py2006-07-20 09:45:57.0 -0400 @@ -162,7 +162,7 @@ def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None): BaseRotatingHandler.__init__(self, filename, 'a', encoding) -self.when = string.upper(when) +self.when = _upper(when) self.backupCount = backupCount # Calculate the real rollover interval, which is just the number of # seconds between rollovers. Also set the filename suffix used when @@ -645,7 +645,7 @@ msg = self.log_format_string % ( self.encodePriority(self.facility, -string.lower(record.levelname)), +_lower(record.levelname)), msg) try: if self.unixsocket: @@ -854,7 +854,7 @@ (GET or POST) logging.Handler.__init__(self) -method = string.upper(method) +method = _upper(method) if method not in [GET, POST]: raise ValueError, method must be GET or POST self.host = host @@ -1007,3 +1007,25 @@ self.flush() self.target = None BufferingHandler.close(self) + +def _upper(s): +A version of upper() that tries to be locale-independent by converting +the string to unicode (which is not subject to case conversion being +locale specific) + +try: +ret = str(unicode(s).upper()) +except NameError: +ret = string.upper(s) +return ret + +def _lower(s): +A version of lower() that tries to be locale-independent by converting +the string to unicode (which is not subject to case conversion being +locale specific) + +try: +ret = str(unicode(s).lower()) +except NameError: +ret = string.lower(s) +return ret ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] logging module broken because of locale
On Thu, Jul 20, 2006 at 06:08:05PM +0200, Martin v. Löwis wrote: Mihai Ibanescu wrote: It's up to Vinay to decide if we want to drop support for 1.5.2 in the module included in newer pythons, or the attached patch would make it work for 1.5.2 as well (as in it's not more broken than before). That still wouldn't work with Python 1.5.2, as that version did not support Unicode at all. Yes, as I said, it won't be more broken than before applying the patch (my first patch was breaking 1.5.2 completely). If people choose to compile python 2.4.3 without unicode support, it won't work either. But if you choose to disable unicode you probably live in a very constrained environment and you may not be affected by the locale bug at all. I agree the patch is not perfect :-) but I think it solves the problem for most people. Misa ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] logging module broken because of locale
On Tue, Jul 18, 2006 at 02:55:56PM -0400, Fred L. Drake, Jr. wrote: On Tuesday 18 July 2006 14:52, Mihai Ibanescu wrote: Unicode might be a perfectly acceptable suggestion for others too. Are we still supporting builds that don't include Unicode? If so, that needs to be considered in a patch as well. Good point. Does the attached patch look reasonable? Thanks, Misa --- Python-2.4.3/Lib/logging/handlers.py.nolocale 2006-07-19 12:15:46.0 -0400 +++ Python-2.4.3/Lib/logging/handlers.py2006-07-19 12:16:14.0 -0400 @@ -44,6 +44,12 @@ DEFAULT_SOAP_LOGGING_PORT = 9023 SYSLOG_UDP_PORT = 514 +# If python was not built with unicode support, use the str function instead +# of the unicode type, and hope locale doesn't break things. + +if not hasattr(__builtins__, 'unicode'): +unicode = str + class BaseRotatingHandler(logging.FileHandler): Base class for handlers that rotate log files at a certain point. @@ -162,7 +168,7 @@ def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None): BaseRotatingHandler.__init__(self, filename, 'a', encoding) -self.when = string.upper(when) +self.when = unicode(when).upper() self.backupCount = backupCount # Calculate the real rollover interval, which is just the number of # seconds between rollovers. Also set the filename suffix used when @@ -642,10 +648,12 @@ We need to convert record level to lowercase, maybe this will change in the future. +We convert it to unicode first, to avoid locale from changing the +meaning of lower() and upper() msg = self.log_format_string % ( self.encodePriority(self.facility, -string.lower(record.levelname)), +unicode(record.levelname).lower()), msg) try: if self.unixsocket: @@ -854,7 +862,7 @@ (GET or POST) logging.Handler.__init__(self) -method = string.upper(method) +method = unicode(method).upper() if method not in [GET, POST]: raise ValueError, method must be GET or POST self.host = host ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] logging module broken because of locale
On Mon, Jul 17, 2006 at 03:39:55PM -0400, Mihai Ibanescu wrote: Hi, This is reported on sourceforge: http://sourceforge.net/tracker/index.php?func=detailaid=1524081group_id=5470atid=105470 I am willing to try and patch the problem, but I'd like to discuss my ideas first. The basic problem is that, in some locale, INFO.lower() != info. So, initializing a dictionary with keys that appear to be lower-case in English is unsafe for other locale. Now, the other problem is, people can choose to initialize the locale any time they want, and can choose to change it too. My initial approach was to create the dictionary with strings to which lower() was explicitly called. But, since priority_names is class-scoped, it gets evaluated at module import time, which is way too early in the game (most likely the locale will be set afterwards). Any thoughts on the proper way to approach this? To follow up on my own email: it looks like, even though in some locale INFO.lower() != info uINFO.lower() == info (at least in the Turkish locale). Is that guaranteed, at least for now (for the current versions of python)? Thanks, Misa ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] logging module broken because of locale
On Tue, Jul 18, 2006 at 10:19:54AM -0700, Guido van Rossum wrote: Alternatively, does info.upper() == INFO everywhere? Not in the Turkish locale :-( # begin /tmp/foo.py import locale locale.setlocale(locale.LC_ALL, '') print info.upper() print info.upper() == INFO # end /tmp/foo.py LANG=tr_TR.UTF-8 python /tmp/foo.py iNFO False Thanks, Misa ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] logging module broken because of locale
On Tue, Jul 18, 2006 at 10:53:23AM -0700, Guido van Rossum wrote: And uinfo.upper()? Yepp, that shows the right thing (at least in the several locales I tested, Turkish included). It's along the lines of uINFO.lower() I was proposing in my second post :-) Misa ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] logging module broken because of locale
On Tue, Jul 18, 2006 at 07:54:28PM +0200, Martin v. Löwis wrote: Mihai Ibanescu wrote: To follow up on my own email: it looks like, even though in some locale INFO.lower() != info uINFO.lower() == info (at least in the Turkish locale). Is that guaranteed, at least for now (for the current versions of python)? It's guaranteed for now; unicode.lower is not locale-aware. OK, should I write a patch for the logging module to convert the string to unicode before applying lower()? So far that seems like the way to go. Maybe this could also be explained in the documentation: http://docs.python.org/lib/node323.html I don't think I've seen it in the locale documentation that locale settings do not affect unicode strings, and that particular page says quote If, when coding a module for general use, you need a locale independent version of an operation that is affected by the locale (such as string.lower(), or certain formats used with time.strftime()), you will have to find a way to do it without using the standard library routine. /quote Unicode might be a perfectly acceptable suggestion for others too. Misa ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] logging module broken because of locale
Hi, This is reported on sourceforge: http://sourceforge.net/tracker/index.php?func=detailaid=1524081group_id=5470atid=105470 I am willing to try and patch the problem, but I'd like to discuss my ideas first. The basic problem is that, in some locale, INFO.lower() != info. So, initializing a dictionary with keys that appear to be lower-case in English is unsafe for other locale. Now, the other problem is, people can choose to initialize the locale any time they want, and can choose to change it too. My initial approach was to create the dictionary with strings to which lower() was explicitly called. But, since priority_names is class-scoped, it gets evaluated at module import time, which is way too early in the game (most likely the locale will be set afterwards). Any thoughts on the proper way to approach this? Thanks, Misa ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
