[Python-Dev] Re: marshal / unmarshal
Fredrik Lundh wrote:
pickle doesn't have the INF=1.0 bug:
import pickle
pickle.loads(pickle.dumps(1e1))
...
ValueError: invalid literal for float(): 1.#INF
import cPickle
cPickle.loads(cPickle.dumps(1e1))
...
ValueError: could not convert string to float
import marshal
marshal.loads(marshal.dumps(1e1))
1.0
should I check in a fix for this?
the code in PyFloat_FromString contains lots of trickery to deal with more or
less
broken literals, and more or less broken C libraries.
unfortunately, and unlike most other functions with similar names,
PyFloat_FromString
takes a Python object, not a char pointer. would it be a good idea to add a
variant
that takes a char*? if so, should PyFloat_FromString use the new function, or
are we
avoiding that kind of refactoring for speed reasons these days?
any opinions?
/F
From yesterday's sprint, we found a smallest-change style fix.
At the least a change like this will catch the unpacking:
in marshal.c (around line 500) in function r_object:
...
case TYPE_FLOAT:
{
char buf[256];
+ char *endp;
double dx;
n = r_byte(p);
if (n == EOF || r_string(buf, (int)n, p) != n) {
PyErr_SetString(PyExc_EOFError,
EOF read where object expected);
return NULL;
}
buf[n] = '\0';
PyFPE_START_PROTECT(atof, return 0)
- dx = PyOS_ascii_atof(buf);
+ dx = PyOS_ascii_strtod(buf, endptr);
PyFPE_END_PROTECT(dx)
+ if buf + n != endptr) {
+ PyErr_SetString(PyExc_ValueError,
+ not all marshalled float text read);
+ return NULL;
+ }
return PyFloat_FromDouble(dx);
}
-- Scott David Daniels
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading (GilState) question
Bob Ippolito [EMAIL PROTECTED] writes: Is there a good reason to *not* call PyEval_InitThreads when using a threaded Python? Well, it depends how expensive ones OS's locking primitives are, I think. There were some numbers posted to the twisted list recently that showed it didn't make a whole lot of difference on some platform or other... I don't have the knowledge or the courage to make that call. Sounds like it would just be easier to implicitly call it during Py_Initialize some day. That might indeed be simpler. Cheers, mwh -- The gripping hand is really that there are morons everywhere, it's just that the Americon morons are funnier than average. -- Pim van Riezen, alt.sysadmin.recovery ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: marshal / unmarshal
Scott David Daniels wrote:
From yesterday's sprint
sprint? I was beginning to wonder why nobody cared about this;
guess I missed the announcement ;-)
At the least a change like this will catch the unpacking:
in marshal.c (around line 500) in function r_object:
PyFPE_START_PROTECT(atof, return 0)
- dx = PyOS_ascii_atof(buf);
+ dx = PyOS_ascii_strtod(buf, endptr);
PyFPE_END_PROTECT(dx)
the PROTECT contents should probably match the function
you're using.
+ if buf + n != endptr) {
+ PyErr_SetString(PyExc_ValueError,
+ not all marshalled float text read);
+ return NULL;
this will fix the problem, sure. I still think it would be cleaner
to reuse the float() semantics, since marshal.dumps uses repr().
to do that, you should use the code in floatobject.c (it wraps
strtod in additional logic designed to take care of various plat-
form quirks).
but nevermind, you have a patch and I don't. if nobody objects,
go ahead and check it in.
/F
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Security capabilities in Python
James Y Knight [EMAIL PROTECTED] writes: On Apr 9, 2005, at 2:13 PM, Michael Hudson wrote: The funniest I know is part of PyPy: def extract_cell_content(c): Get the value contained in a CPython 'cell', as read through the func_closure of a function object. # yuk! this is all I could come up with that works in Python 2.2 too class X(object): def __eq__(self, other): self.other = other x = X() x_cell, = (lambda: x).func_closure x_cell == c return x.other It would be unfortunate for PyPy (and IMHO, very un-pythonic) if this process became impossible. It would be quite fortunate if you didn't have to do all that, and cell just had a value attribute, though. Indeed. The 2.2 compatibility issue remains, though. Cheers, mwh -- Presumably pronging in the wrong place zogs it. -- Aldabra Stoddart, ucam.chat ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Security capabilities in Python
It may be really hard to get it right, unless we are overlooking some simple solution.I disagree that we should just use OS protections.The
reason I am interested in Pythonic protection is because it is so much
more powerful than OS protections.The capability model is
much more powerful than the ACL model used by all OS's these days, and
allows for interesting security concepts.What about implementing the facet in C? This could avoid the class of problems you have just mentioned.On Apr 9, 2005 2:02 PM, James Y Knight [EMAIL PROTECTED] wrote: On Apr 9, 2005, at 5:37 PM, Ka-Ping Yee wrote: Let me know if you figure out how to defeat that. You can protect against this, too, but it does show that it's *really* hard to get restricting code right...I'm of the opinion that it's not really worth it -- you should just use OS protections. untrusted_module.py: class foostr(str): def __eq__(self, other):return True def have_at_it(immutable_facet, readonly_facet):getattr(immutable_facet, foostr('append'))(5)print immutable_facet James___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: marshal / unmarshal
marshal shouldn't be representing doubles as decimal strings to begin with. All code for (de)serialing C doubles should go thru _PyFloat_Pack8() and _PyFloat_Unpack8(). cPickle (proto = 1) and struct (std mode) already do; marshal is the oddball. But as the docs (floatobject.h) for these say: ... * Bug: What this does is undefined if x is a NaN or infinity. * Bug: -0.0 and +0.0 produce the same string. */ PyAPI_FUNC(int) _PyFloat_Pack4(double x, unsigned char *p, int le); PyAPI_FUNC(int) _PyFloat_Pack8(double x, unsigned char *p, int le); ... * Bug: What this does is undefined if the string represents a NaN or * infinity. */ PyAPI_FUNC(double) _PyFloat_Unpack4(const unsigned char *p, int le); PyAPI_FUNC(double) _PyFloat_Unpack8(const unsigned char *p, int le); ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Re: marshal / unmarshal
Tim Peters wrote: marshal shouldn't be representing doubles as decimal strings to begin with. All code for (de)serialing C doubles should go thru _PyFloat_Pack8() and _PyFloat_Unpack8(). cPickle (proto = 1) and struct (std mode) already do; marshal is the oddball. is changing the marshal format really the right thing to do at this point? /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: marshal / unmarshal
Tim Peters [EMAIL PROTECTED] writes:
marshal shouldn't be representing doubles as decimal strings to begin
with. All code for (de)serialing C doubles should go thru
_PyFloat_Pack8() and _PyFloat_Unpack8(). cPickle (proto = 1) and
struct (std mode) already do; marshal is the oddball.
But as the docs (floatobject.h) for these say:
...
* Bug: What this does is undefined if x is a NaN or infinity.
* Bug: -0.0 and +0.0 produce the same string.
*/
PyAPI_FUNC(int) _PyFloat_Pack4(double x, unsigned char *p, int le);
PyAPI_FUNC(int) _PyFloat_Pack8(double x, unsigned char *p, int le);
...
* Bug: What this does is undefined if the string represents a NaN or
* infinity.
*/
PyAPI_FUNC(double) _PyFloat_Unpack4(const unsigned char *p, int le);
PyAPI_FUNC(double) _PyFloat_Unpack8(const unsigned char *p, int le);
OTOH, the implementation has this comment:
/*
* _PyFloat_{Pack,Unpack}{4,8}. See floatobject.h.
*
* TODO: On platforms that use the standard IEEE-754 single and double
* formats natively, these routines could simply copy the bytes.
*/
Doing that would fix these problems, surely?[1]
The question, of course, is how to tell. I suppose one could jsut do
it unconditionally and wait for one of the three remaining VAX
users[2] to compile Python 2.5 and then notice.
More conservatively, one could just do this on Windows, linux/most
architectures and Mac OS X.
Cheers,
mwh
[1] I'm slighyly worried about oddball systems that do insane things
with the FPU by default -- but don't think the mooted change would
make things any worse.
[2] Exaggeration, I realize -- but how many non 754 systems are out
there? How many will see Python 2.5?
--
If you give someone Fortran, he has Fortran.
If you give someone Lisp, he has any language he pleases.
-- Guy L. Steele Jr, quoted by David Rush in comp.lang.scheme.scsh
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: marshal / unmarshal
Michael I suppose one could jsut do it unconditionally and wait for one Michael of the three remaining VAX users[2] to compile Python 2.5 and Michael then notice. You forgot the two remaining CRAY users. Since their machines are so much more powerful than VAXen, they have much more influence over Python development. wink Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading (GilState) question
On Apr 10, 2005, at 11:22 AM, Michael Hudson wrote:
Bob Ippolito [EMAIL PROTECTED] writes:
Is there a good reason to *not* call PyEval_InitThreads when using a
threaded Python?
Well, it depends how expensive ones OS's locking primitives are, I
think. There were some numbers posted to the twisted list recently
that showed it didn't make a whole lot of difference on some platform
or other... I don't have the knowledge or the courage to make that
call.
Sounds like it would just be easier to implicitly call it during
Py_Initialize some day.
That might indeed be simpler.
Here's the numbers. It looks like something changed between python 2.2
and 2.3 that made calling PyEval_InitThreads a lot less expensive. So,
it doesn't seem to make a whole lot of difference on recent versions of
Python.
Three test programs:
${PYTHON} -c 'import pystone, time; print pystone.pystones(20)'
${PYTHON} -c 'import thread, pystone, time; print
pystone.pystones(20)'
${PYTHON} -c 'import thread, pystone, time;
thread.start_new_thread(lambda: time.sleep(1), ()); print
pystone.pystones(20)'
All tests run using the same copy of pystone.
System 1: RH73, dual 3GHz Xeon
[GCC 2.96 2731 (Red Hat Linux 7.3 2.96-110)]
Python 1.5.2 (#1, Apr 3 2002, 18:16:26)
(8.15, 24540)
(8.28, 24155)
(12.78, 15649)
Python 2.2.2 (#1, Jul 23 2003, 13:47:48)
(6.32, 31646)
(6.27, 31898)
(11.1, 18018)
Python 2.4.1 (#1, Apr 4 2005, 17:19:27)
(4.60, 43478)
(4.61, 43384)
(4.74, 42194)
System 2, FC3/64, dual 2.4GHz athlon 64.
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)]
Python 2.3.4 (#1, Oct 26 2004, 16:45:38)
(3.84, 52083)
(3.80, 52632)
(3.98, 50251)
Python 2.4.1 (#1, Apr 10 2005, 15:47:53)
(3.09, 64725)
(3.08, 64935)
(3.26, 61350)
Python 2.4.1 (#1, Apr 1 2005, 16:45:07)
*compiled in 32 bit mode*
(3.35, 59701)
(3.42, 58480)
(3.57, 56022)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading (GilState) question
James Y Knight [EMAIL PROTECTED] writes: On Apr 10, 2005, at 11:22 AM, Michael Hudson wrote: Bob Ippolito [EMAIL PROTECTED] writes: Is there a good reason to *not* call PyEval_InitThreads when using a threaded Python? Well, it depends how expensive ones OS's locking primitives are, I think. There were some numbers posted to the twisted list recently that showed it didn't make a whole lot of difference on some platform or other... I don't have the knowledge or the courage to make that call. Sounds like it would just be easier to implicitly call it during Py_Initialize some day. That might indeed be simpler. Here's the numbers. It looks like something changed between python 2.2 and 2.3 that made calling PyEval_InitThreads a lot less expensive. So, it doesn't seem to make a whole lot of difference on recent versions of Python. Thanks. I see similar results for 2.3 and 2.4 on OS X (don't have 2.2 here). It's very much a guess, but could this patch: [ 525532 ] Add support for POSIX semaphores be the one to thank? Cheers, mwh -- Now this is what I don't get. Nobody said absolutely anything bad about anything. Yet it is always possible to just pull random flames out of ones ass. -- http://www.advogato.org/person/vicious/diary.html?start=60 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading (GilState) question
On Apr 10, 2005, at 2:48 PM, Michael Hudson wrote: James Y Knight [EMAIL PROTECTED] writes: On Apr 10, 2005, at 11:22 AM, Michael Hudson wrote: Bob Ippolito [EMAIL PROTECTED] writes: Is there a good reason to *not* call PyEval_InitThreads when using a threaded Python? Well, it depends how expensive ones OS's locking primitives are, I think. There were some numbers posted to the twisted list recently that showed it didn't make a whole lot of difference on some platform or other... I don't have the knowledge or the courage to make that call. Sounds like it would just be easier to implicitly call it during Py_Initialize some day. That might indeed be simpler. Here's the numbers. It looks like something changed between python 2.2 and 2.3 that made calling PyEval_InitThreads a lot less expensive. So, it doesn't seem to make a whole lot of difference on recent versions of Python. Thanks. I see similar results for 2.3 and 2.4 on OS X (don't have 2.2 here). It's very much a guess, but could this patch: [ 525532 ] Add support for POSIX semaphores be the one to thank? No, Mac OS X doesn't implement POSIX semaphores. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: marshal / unmarshal
[mwh]
OTOH, the implementation has this comment:
/*
* _PyFloat_{Pack,Unpack}{4,8}. See floatobject.h.
*
* TODO: On platforms that use the standard IEEE-754 single and double
* formats natively, these routines could simply copy the bytes.
*/
Doing that would fix these problems, surely?[1]
The 754 standard doesn't say anything about how the difference between
signaling and quiet NaNs is represented. So it's possible that a qNaN
on one box would look like an sNaN on a different box, and vice
versa. But since most people run with all FPU traps disabled, and
Python doesn't expose a way to read the FPU status flags, they
couldn't tell the difference.
Copying bytes works perfectly for all other cases (signed zeroes,
non-zero finites, infinities), because their representations are
wholly defined, although it's possible that a subnormal on one box
will be treated like a zero (with the same sign) on a
partially-conforming box.
[1] I'm slighyly worried about oddball systems that do insane things
with the FPU by default -- but don't think the mooted change would
make things any worse.
Sorry, don't know what that means.
The question, of course, is how to tell.
Store a few small doubles at module initialization time and stare at
their bits. That's enough to settle whether a 754 format is in use,
and, if it is, whether it's big-endian or little-endian.
...
[2] Exaggeration, I realize -- but how many non 754 systems are out
there? How many will see Python 2.5?
No idea here. The existing pack routines strive to do a good job of
_creating_ an IEEE-754-format representation regardless of platform
representation. I assume that code would still be present, so
oddball platforms would be left no worse off than they are now.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading (GilState) question
Bob Ippolito [EMAIL PROTECTED] writes: On Apr 10, 2005, at 2:48 PM, Michael Hudson wrote: James Y Knight [EMAIL PROTECTED] writes: Here's the numbers. It looks like something changed between python 2.2 and 2.3 that made calling PyEval_InitThreads a lot less expensive. So, it doesn't seem to make a whole lot of difference on recent versions of Python. Thanks. I see similar results for 2.3 and 2.4 on OS X (don't have 2.2 here). It's very much a guess, but could this patch: [ 525532 ] Add support for POSIX semaphores be the one to thank? No, Mac OS X doesn't implement POSIX semaphores. Well, does OS X show the same effect between 2.2 and 2.3? I don't have a 2.2 on OS X any more, I was just talking about James' results on linux. Cheers, mwh -- Slim Shady is fed up with your shit, and he's going to kill you. -- Eminem, Public Service Announcement 2000 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threading (GilState) question
On Apr 10, 2005, at 4:08 PM, Michael Hudson wrote: Bob Ippolito [EMAIL PROTECTED] writes: On Apr 10, 2005, at 2:48 PM, Michael Hudson wrote: James Y Knight [EMAIL PROTECTED] writes: Here's the numbers. It looks like something changed between python 2.2 and 2.3 that made calling PyEval_InitThreads a lot less expensive. So, it doesn't seem to make a whole lot of difference on recent versions of Python. Thanks. I see similar results for 2.3 and 2.4 on OS X (don't have 2.2 here). It's very much a guess, but could this patch: [ 525532 ] Add support for POSIX semaphores be the one to thank? No, Mac OS X doesn't implement POSIX semaphores. Well, does OS X show the same effect between 2.2 and 2.3? I don't have a 2.2 on OS X any more, I was just talking about James' results on linux. I don't have 2.2 on OS X any more, either. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: marshal / unmarshal
On Apr 10, 2005, at 13:44, Skip Montanaro wrote: Michael I suppose one could jsut do it unconditionally and wait for one Michael of the three remaining VAX users[2] to compile Python 2.5 and Michael then notice. You forgot the two remaining CRAY users. Since their machines are so much more powerful than VAXen, they have much more influence over Python development. wink The latest ads I've seen from Cray were touting AMD-64 processors anyway...;-) Alex ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
