Re: [Python-Dev] Segfault
On Tue, 2007-08-21 at 09:14 -0700, Neal Norwitz wrote: > The patch is insufficient to prevent all types of crashes that occur > when accessing a file from 2 threads (closing in one and doing > whatever in another). You are right. I wouldn't go so far to say the file object thread-unsafe, but it certainly has lurking bugs with calling close while other threads are running. BTW your mail doesn't seem to contain the actual patch. > Almost every place that accesses f_fp is a problem. I think we need a reliable mechanism to prevent file_close messing with f_fp while other operations are being performed. Since each FILE has an internal lock associated with it, flockfile could be used to lock out the sections that access f_fp (falling back to a mutex in PyFileObject on platforms without flockfile) without affecting other threads that operate on other FILE's. It's questionnable whether such an overhaul would pay off, given the Py3k rewrite of the IO subsystem. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Segfault
On Wed, 2007-08-22 at 10:35 +0200, Hrvoje Nikšić wrote: > I think we need a reliable mechanism to prevent file_close messing with > f_fp while other operations are being performed. Since each FILE has an > internal lock associated with it, flockfile could be used to lock out > the sections that access f_fp (falling back to a mutex in PyFileObject > on platforms without flockfile) without affecting other threads that > operate on other FILE's. Scrap this suggestion, it won't work as written. flockfile must not be called after the FILE is deallocated, which is exactly what fclose does. The mutex would need to be in PyFileObject itself. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How to interpret get_code from PEP 302?
Greg Ewing wrote: > Paul Moore wrote: >> What the sentence you quote >> is trying to say is that if there's a need to compile source, the >> get_code method must do this on behalf of the caller - it can't return >> None and expect the caller to try get_source and compile it manually. > > Why not simply say that it must return a code object? > All the rest follows from that. The method is allowed to return None if a code object genuinely isn't available (which happens when the module is a builtin or extension module). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Documentation switch imminent
Georg Brandl wrote: > Okay, I uploaded a new version without JavaScript and with hidden permalink > markers. Very nice. I've compared this version and the 2.4 CHM a little and while mostly similar, there are several small improvements I appreciate. One other minor issue, when viewing the "random" module docs, it looks like two or more lines overlap in the footer (scroll all the way down). -- Benji York http://benjiyork.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More on server-side SSL support
> getpeercert() -- analogue to "getpeeraddr", but returns cert details
This would return three kinds of values:
No certificate received --> None
Certificate received but not validated --> {}
Certificate received and validated --> { full details }
Bill
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More on server-side SSL support
For those of you following along at home, there's now a patch at http://bill.janssen.org/python/ssl-update-diff which applies against the current trunk. Working code, though I still need to tweak some import statements for backwards compatibility. I've started updating the test suite, see Lib/test_ssl.py. I'd appreciate review and comments. Note that the suggested use of the newer code is to import ssl, socket conn = ssl.sslsocket(socket.socket()) Using the older import socket conn = socket.ssl(socket.socket()) still works for backwards compatibility (same restrictions: client-side only, no verification of server cert, SSLv23). However, the object returned from this call is not the older C-implemented SSL object, but rather an instance of ssl.sslsocket (which supports the same read() and write() calls). Should I return the C SSL object, instead? Bill > > I view TLS as a wrapper around / layer on top of TCP, and so I think the > > API should look like, as well. > > I think Martin raises a valid point here, which should at least be > discussed more thoroughly. Should there be an "SSL socket", which is > just like a regular socket? Does that really provide any additional > functionality to anyone? Most apps and classes that use TCP sockets > wrap the socket with socket._fileobject() to get "read" and "write", > anyway -- can't they just wrap it with socket.ssl instead? > > Perhaps in the sprint, I should just concentrate on widening the > "socket.ssl" interface a bit, and improving the functionality of the > SSLObject a bit. > > Suggested improvements: > > * Allow server-side operation. > > * Allow specification of particular SSL protocol version. > > * Allow certificate validation. This is a bit tricky; typically > certs are validated against some database of root certificates, so you > need a whole infrastructure to maintain that database. Currently, we > don't have one, so no certs can be validated. We could add a switch > to allow auto-validation of self-signed certs pretty easily. I could > add a parameter to the SSLObject constructor which would be a filepath > for a file full of root certs (see SSL_CTX_load_verify_locations(3ssl)). > > * Add a method to retrieve the other side's certificate info. What's > a good format for the "notBefore" and "notAfter" dates? The simplest > thing to use is the string formatting OpenSSL provides, which is > always like "Sep 29 16:38:04 2006 GMT", which can easily be parsed > by the time.strptime() function if the user wants something else. > On the other hand, figuring out how to use strptime() is always a > pain, so providing a convenience function wouldn't be a terrible idea. > > * Add a shutdown() method to stop using SSL on the underlying socket > without closing the socket. > > * Make SSLObject conform to the Raw I/O API in PEP 3116. This one is > interesting; what should close() do? Close the underlying socket? Or > just do an SSL shutdown? > > Bill > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/janssen%40parc.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] AtheOS/Syllable
The Syllable folks apparently still want to support Python, so I'm going to keep that support in there, switching "atheos" for "syllable" where it occurs. I have a patch against 2.5.1 from the Syllable folks. I'll see if I can make it apply to the trunk and py3k branches in a semi-clean fashion. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Strange behavior of subprocess.Popen._get_handles under Windows
For a long time I was surprised why if I have a script testme.py:
import subprocess
subprocess.call("echo Something", shell=True)
and I try to execute it like this:
python testme.py >testme.txt
I get the output:
The handle is invalid.
Strange failures randomly happened with different programs, so I
thought maybe this was intended (mis)behavior, and I found that I can
workaround this by explicitly specifying std* handles:
subprocess.call(, stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr)
I lived with this until I understood that if something substitutes
sys.std* with a proxy (examples: running under IDLE or wxPython, or
when there is a custom logger on stdout/stderr, etc), this will no
longer work:
File "C:\Programs\Python25\Lib\subprocess.py", line 698, in _get_handles
p2cread = msvcrt.get_osfhandle(stdin.fileno())
AttributeError: fileno
Now I finally found that my problem are these two lines in subprocess.py:
if stdin is None and stdout is None and stderr is None:
return (None, None, None, None, None, None)
These lines add an interesting quirk: if I explicitly specify any
single channel (i.e. stdout=1) the problem suddenly disappears, and if
I just comment these lines altogether my problem vanishes completely!
(i.e. output redirection works absolutely well)
Further investigations showed that it seems to be some strange OS
quirk/bug, i.e. this behavior is caused when STARTF_USESTDHANDLES is
not used, and crt's spawnl* behaves incorrectly too:
os.spawnl(r"C:\WINDOWS\system32\cmd.exe",
r"C:\WINDOWS\system32\cmd.exe /c echo Something")
So the question is should these two lines be removed to workaround this OS bug?
To my thinking, this will not change behavior when any std* arguments
are passed to Popen, and the only case when it kicks in is when no
std* arguments are specified and the resulting side effects are
previously failing cases start working correctly. Or am I wrong here,
are there any side effects of using STARTF_USESTDHANDLES that I'm
missing here?
Best regards,
Alexey.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange behavior of subprocess.Popen._get_handles under Windows
> Now I finally found that my problem are these two lines in > subprocess.py: > > if stdin is None and stdout is None and stderr is None: > return (None, None, None, None, None, None) > > These lines add an interesting quirk: if I explicitly specify any > single channel (i.e. stdout=1) the problem suddenly disappears, and if > I just comment these lines altogether my problem vanishes completely! > (i.e. output redirection works absolutely well) > > Further investigations showed that it seems to be some strange OS > quirk/bug, I'm not quite with you here - what strange OS bug do you think you have found? I expect that such a bug would be well documented somewhere, even if not directly by MS. > To my thinking, this will not change behavior when any std* arguments > are passed to Popen, and the only case when it kicks in is when no > std* arguments are specified and the resulting side effects are > previously failing cases start working correctly. Or am I wrong here, > are there any side effects of using STARTF_USESTDHANDLES that I'm > missing here? MSDN documents that without that flag, hStdInput will be "the keyboard buffer" while the output handles will default to "the console window's buffer" - which sounds significantly different to your expectation that the existing handles will be used as the default (unless I misunderstand). Sadly, your mail isn't clear enough for me to be sure about what semantics you are seeing and exactly what you expect, and I don't have time to experiment. Maybe a clear indication of the OS bug you are referring to, and some complete examples which demonstrate the problems you are having would help. Cheers, Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Avoiding cascading test failures
When I was fixing tests failing in the py3k branch, I found the number
duplicate failures annoying. Often, a single bug, in an important
method or function, caused a large number of testcase to fail. So, I
thought of a simple mechanism for avoiding such cascading failures.
My solution is to add a notion of dependency to testcases. A typical
usage would look like this:
@depends('test_getvalue')
def test_writelines(self):
...
memio.writelines([buf] * 100)
self.assertEqual(memio.getvalue(), buf * 100)
...
Here, running the test is pointless if test_getvalue fails. So by
making test_writelines depends on the success of test_getvalue, we can
ensure that the report won't be polluted with unnecessary failures.
Also, I believe this feature will lead to more orthogonal tests, since
it encourages the user to write smaller test with less dependencies.
I wrote an example implementation (included below) as a proof of
concept. If the idea get enough support, I will implement it and add
it to the unittest module.
-- Alexandre
class CycleError(Exception):
pass
class TestCase:
def __init__(self):
self.graph = {}
tests = [x for x in dir(self) if x.startswith('test')]
for testname in tests:
test = getattr(self, testname)
if hasattr(test, 'deps'):
self.graph[testname] = test.deps
else:
self.graph[testname] = set()
def run(self):
graph = self.graph
toskip = set()
msgs = []
while graph:
# find tests without any pending dependencies
source = [test for test, deps in graph.items() if not deps]
if not source:
raise CycleError
for testname in source:
if testname in toskip:
msgs.append("%s... skipped" % testname)
resolvedeps(graph, testname)
del graph[testname]
continue
test = getattr(self, testname)
try:
test()
except AssertionError:
toskip.update(getrevdeps(graph, testname))
msgs.append("%s... failed" % testname)
except:
toskip.update(getrevdeps(graph, testname))
msgs.append("%s... error" % testname)
else:
msgs.append("%s... ok" % testname)
finally:
resolvedeps(graph, testname)
del graph[testname]
for msg in sorted(msgs):
print(msg)
def getrevdeps(graph, testname):
"""Return the reverse depencencies of a test"""
rdeps = set()
for x in graph:
if testname in graph[x]:
rdeps.add(x)
if rdeps:
# propagate depencencies recursively
for x in rdeps.copy():
rdeps.update(getrevdeps(graph, x))
return rdeps
def resolvedeps(graph, testname):
for test in graph:
if testname in graph[test]:
graph[test].remove(testname)
def depends(*args):
def decorator(test):
if hasattr(test, 'deps'):
test.deps.update(args)
else:
test.deps = set(args)
return test
return decorator
class MyTest(TestCase):
@depends('test_foo')
def test_nah(self):
pass
@depends('test_bar', 'test_baz')
def test_foo(self):
pass
@depends('test_tin')
def test_bar(self):
self.fail()
def test_baz(self):
self.error()
def test_tin(self):
pass
def error(self):
raise ValueError
def fail(self):
raise AssertionError
if __name__ == '__main__':
t = MyTest()
t.run()
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Segfault
On 8/22/07, Hrvoje Nikšić <[EMAIL PROTECTED]> wrote:
> On Tue, 2007-08-21 at 09:14 -0700, Neal Norwitz wrote:
> > The patch is insufficient to prevent all types of crashes that occur
> > when accessing a file from 2 threads (closing in one and doing
> > whatever in another).
>
> You are right. I wouldn't go so far to say the file object
> thread-unsafe, but it certainly has lurking bugs with calling close
> while other threads are running. BTW your mail doesn't seem to contain
> the actual patch.
Sorry about that. I thought something was up when gmail didn't
automatically upload it while I was writing the message. It looks
like it took this time. (The attached patch isn't complete, but
catches most cases.)
n
Index: Objects/fileobject.c
===
--- Objects/fileobject.c (revision 57244)
+++ Objects/fileobject.c (working copy)
@@ -439,13 +439,16 @@
{
int sts = 0;
if (f->f_fp != NULL) {
+ /* Invalidate f->f_fp *before* closing, so we don't attempt
+ to close it twice in separate threads. */
+ FILE *tmp_fp = f->f_fp;
+ f->f_fp = NULL;
if (f->f_close != NULL) {
Py_BEGIN_ALLOW_THREADS
errno = 0;
- sts = (*f->f_close)(f->f_fp);
+ sts = (*f->f_close)(tmp_fp);
Py_END_ALLOW_THREADS
}
- f->f_fp = NULL;
}
PyMem_Free(f->f_setbuf);
f->f_setbuf = NULL;
@@ -539,7 +542,7 @@
file_seek(PyFileObject *f, PyObject *args)
{
int whence;
- int ret;
+ int ret = 0;
Py_off_t offset;
PyObject *offobj;
@@ -560,8 +563,11 @@
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = _portable_fseek(f->f_fp, offset, whence);
+ if (f->f_fp != NULL)
+ ret = _portable_fseek(f->f_fp, offset, whence);
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (ret != 0) {
PyErr_SetFromErrno(PyExc_IOError);
@@ -580,8 +586,8 @@
{
Py_off_t newsize;
PyObject *newsizeobj = NULL;
- Py_off_t initialpos;
- int ret;
+ Py_off_t initialpos = 0;
+ int ret = 0;
if (f->f_fp == NULL)
return err_closed();
@@ -597,8 +603,11 @@
*/
Py_BEGIN_ALLOW_THREADS
errno = 0;
- initialpos = _portable_ftell(f->f_fp);
+ if (f->f_fp != NULL)
+ initialpos = _portable_ftell(f->f_fp);
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (initialpos == -1)
goto onioerror;
@@ -625,8 +634,11 @@
*/
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = fflush(f->f_fp);
+ if (f->f_fp != NULL)
+ ret = fflush(f->f_fp);
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (ret != 0)
goto onioerror;
@@ -639,30 +651,40 @@
/* Have to move current pos to desired endpoint on Windows. */
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = _portable_fseek(f->f_fp, newsize, SEEK_SET) != 0;
+ if (f->f_fp != NULL)
+ ret = _portable_fseek(f->f_fp, newsize, SEEK_SET) != 0;
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (ret)
goto onioerror;
/* Truncate. Note that this may grow the file! */
Py_BEGIN_ALLOW_THREADS
errno = 0;
- hFile = (HANDLE)_get_osfhandle(fileno(f->f_fp));
- ret = hFile == (HANDLE)-1;
- if (ret == 0) {
- ret = SetEndOfFile(hFile) == 0;
- if (ret)
-errno = EACCES;
+ if (f->f_fp != NULL) {
+ hFile = (HANDLE)_get_osfhandle(fileno(f->f_fp));
+ ret = hFile == (HANDLE)-1;
+ if (ret == 0) {
+ret = SetEndOfFile(hFile) == 0;
+if (ret)
+ errno = EACCES;
+ }
}
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (ret)
goto onioerror;
}
#else
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = ftruncate(fileno(f->f_fp), newsize);
+ if (f->f_fp != NULL)
+ ret = ftruncate(fileno(f->f_fp), newsize);
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (ret != 0)
goto onioerror;
#endif /* !MS_WINDOWS */
@@ -670,8 +692,11 @@
/* Restore original file position. */
Py_BEGIN_ALLOW_THREADS
errno = 0;
- ret = _portable_fseek(f->f_fp, initialpos, SEEK_SET) != 0;
+ if (f->f_fp != NULL)
+ ret = _portable_fseek(f->f_fp, initialpos, SEEK_SET) != 0;
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (ret)
goto onioerror;
@@ -688,14 +713,17 @@
static PyObject *
file_tell(PyFileObject *f)
{
- Py_off_t pos;
+ Py_off_t pos = 0;
if (f->f_fp == NULL)
return err_closed();
Py_BEGIN_ALLOW_THREADS
errno = 0;
- pos = _portable_ftell(f->f_fp);
+ if (f->f_fp != NULL)
+ pos = _portable_ftell(f->f_fp);
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (pos == -1) {
PyErr_SetFromErrno(PyExc_IOError);
clearerr(f->f_fp);
@@ -727,14 +755,17 @@
static PyObject *
file_flush(PyFileObject *f)
{
- int res;
+ int res = 0;
if (f->f_fp == NULL)
return err_closed();
Py_BEGIN_ALLOW_THREADS
errno = 0;
- res = fflush(f->f_fp);
+ if (f->f_fp != NULL)
+ res = fflush(f->f_fp);
Py_END_ALLOW_THREADS
+ if (f->f_fp == NULL)
+ return err_closed();
if (res != 0) {
PyErr_SetFromErrno(PyExc_IOError);
clearerr(f->f_
Re: [Python-Dev] Strange behavior of subprocess.Popen._get_handles under Windows
On 8/23/07, Mark Hammond <[EMAIL PROTECTED]> wrote:
> > Further investigations showed that it seems to be some strange OS
> > quirk/bug,
> I'm not quite with you here - what strange OS bug do you think you have
> found? I expect that such a bug would be well documented somewhere, even if
> not directly by MS.
[...]
> MSDN documents that without that flag, hStdInput will be "the keyboard
> buffer" while the output handles will default to "the console window's
> buffer" - which sounds significantly different to your expectation that the
> existing handles will be used as the default (unless I misunderstand).
> Sadly, your mail isn't clear enough for me to be sure about what semantics
> you are seeing and exactly what you expect, and I don't have time to
> experiment. Maybe a clear indication of the OS bug you are referring to,
> and some complete examples which demonstrate the problems you are having
> would help.
Hmm, sorry if I wasn't clear. Here's a better example:
1.c:
#include
#include
int main(int argc, char** argv)
{
HANDLE hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);
fprintf(stderr, "hStdOut: %08X\n", hStdOut);
DWORD dwWritten;
WriteFile(hStdOut, "Something", 9, &dwWritten, NULL);
return 0;
}
1.py:
import subprocess
subprocess.call(["1.exe"])
in cmd.exe shell:
C:\>1.exe
hStdOut: 0007
Something
C:\>1.exe>1.txt
hStdOut: 06E0
(1.txt file now contains string "Something")
C:\>1.py
hStdOut: 0007
Something
C:\>1.py>1.txt
hStdOut: 0004
(1.txt file is now completely empty, i.e. hStdOut is invalid)
If what you say was true and stdout would default to console window's
buffer, then I would at least see "Something" in the console. On the
contrary output just stops working at all.
And what I expect (and always expected) is output redirection to be
inherited by child processes. The best example is when I'm writing a
batch file:
dosomething.exe /something
dosomething2.exe /somethingmore
and then execute it as batchfile.cmd>somefile.txt I always get this
behavior, i.e. all standard output (except standard error) is
redirected to somefile.txt. I'm sure on *nixes it's all the same as
well. So what documentation for STARTUPINFO specifies is already very
suspicious and uncommon, while in reality it doesn't even work as
specified!
Best regards,
Alexey.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange behavior of subprocess.Popen._get_handles under Windows
> C:\>1.py > hStdOut: 0007 > Something > > C:\>1.py>1.txt > hStdOut: 0004 > > (1.txt file is now completely empty, i.e. hStdOut is invalid) Hmm, now I see that maybe that's where I was wrong. When I used `python 1.py>1.txt' it suddenly started working as expected. :-/ Must be some strange difference with how HKCR entries are parsed and then executed... Best regards, Alexey. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange behavior of subprocess.Popen._get_handles under Windows
> > C:\>1.py > > hStdOut: 0007 > > Something > > > > C:\>1.py>1.txt > > hStdOut: 0004 > > > > (1.txt file is now completely empty, i.e. hStdOut is invalid) > > Hmm, now I see that maybe that's where I was wrong. When I used > `python 1.py>1.txt' it suddenly started working as expected. :-/ Must > be some strange difference with how HKCR entries are parsed and then > executed... I think it is more about how the shell (ie, cmd.exe by default) (mis-)handles the command-line. Cheers, Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
