Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
Cameron Simpson wrote: On 16Jun2009 02:18, MRAB pyt...@mrabarnett.plus.com wrote: My itch is that peek() _feels_ like it should be look into the buffer but actually can block and/or change the buffer. Can block, but not if you don't want it too. You might just want to see what, if anything, is currently available, up to n bytes. Am I missing something? In the face of an _empty_ buffer (which I can't tell from outside) how do I prevent peek() blocking? More generally, if I go peek(n) and if n bytes_in_buffer_right_now and the raw stream would block if a raw read is done? My concerns would go away if I could probe the buffer content size; then I could ensure peek(n) chose n = the content size. If that's not enough, my problem - I can choose to read-and-block or go away and come back later. I was thinking along the lines of: def peek(self, size=None, block=True) If 'block' is True then return 'size' bytes, unless the end of the file/stream is reached; if 'block' is False then return up to 'size' bytes, without blocking. The blocking form might impose a limit to how much can be peeked (the maximum size of the buffer), or it might enlarge the buffer as necessary. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoiding file descriptors leakage in subprocess.Popen()
On Tue, 16 Jun 2009 01:20:53 pm Cameron Simpson wrote: I don't think all pythons do immediate ref-counted GC. Jython and IronPython don't. I don't know about PyPy, CLPython, Unladen Swallow, etc. -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoiding file descriptors leakage in subprocess.Popen()
Steven D'Aprano wrote: On Tue, 16 Jun 2009 01:20:53 pm Cameron Simpson wrote: I don't think all pythons do immediate ref-counted GC. Jython and IronPython don't. I don't know about PyPy, CLPython, Unladen Swallow, etc. PyPy doesn't, Unladen Swallow won't. Michael -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
Cameron Simpson wrote: Indeed, though arguably read1() is a lousy name too, on the same basis. My itch is that peek() _feels_ like it should be look into the buffer but actually can block and/or change the buffer. I guess all the buffer operations should be transparent to the user if he wants it to be like that, since not so many people want to have a tight control over this kind of detail. I think of peek() as an operation that allows me to peek what's going to show up in the future without affecting further read()s. This kind of behavior is expected by users without prior knowledge of the inner workings of buffered IO. So, if an user _really_ wants to take a look at what's to come without affecting the buffer, we could allow that by doing something like this: peek(5, change_buffer=False) This is an alternative to the peek0(). But I am ok wih the peek0() too. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
MRAB wrote: I was thinking along the lines of: def peek(self, size=None, block=True) I think this is fine too. :) If 'block' is True then return 'size' bytes, unless the end of the file/stream is reached; if 'block' is False then return up to 'size' bytes, without blocking. The blocking form might impose a limit to how much can be peeked (the maximum size of the buffer), or it might enlarge the buffer as necessary. I guess the limit wouldn't be a problem to someone that chose to block further reads. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoiding file descriptors leakage in subprocess.Popen()
On Tue, Jun 16, 2009 at 1:21 PM, Michael Foordfuzzy...@voidspace.org.uk wrote: Steven D'Aprano wrote: On Tue, 16 Jun 2009 01:20:53 pm Cameron Simpson wrote: I don't think all pythons do immediate ref-counted GC. Jython and IronPython don't. I don't know about PyPy, CLPython, Unladen Swallow, etc. PyPy doesn't, Unladen Swallow won't. Also most Lisp implementations, thus CLPython, use a tracing GC. - Willem ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] SSL Certificate Validation
Hi all, I have a few questions about validating SSL certificates. From what I gather, this validation occurs in the OpenSSL code called from _ssl.c. Is this correct? Also, I have looked through the docs and code, but haven't been able to figure out exactly what is included in certificate validation. Is it just validating the chain? Does it check the NotBefore and NotAfter dates? Does it check that the host the socket is connected to is the same as what's given in the CN field in the certificate? Where I'm going with this is I think all this checking needs to be part of certificate validation in the ssl module. If it isn't yet, I'd be happy to work on a patch for it. Please let me know what you think. Thanks! -Devin Cook ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Avoiding file descriptors leakage in subprocess.Popen()
Cameron Simpson c...@zip.com.au wrote: On 14Jun2009 16:42, Mark Seaborn m...@mythic-beasts.com wrote: | I use a convenience function like this, so that GC takes care of the FDs: | | def make_pipe(): | read_fd, write_fd = os.pipe() | return os.fdopen(read_fd, r), os.fdopen(write_fd, w) Not guarrenteed to be timely. The try/except at least closes things as control passes out of the relevant scope. I don't think all pythons do immediate ref-counted GC. Yep. I don't mind closing FDs explicitly when it's easy to do so in a try..finally, but it's not always simple. There are two different problems with non-prompt closing of FDs: * Whether an FD has been closed is sometimes externally observable. e.g. Pipes and sockets notify the other end of the connection. Open file and directory FDs prevent filesystems from being unmounted. * FDs use up space in the process's FD table. The second problem could be dealt with by running the GC when we get EMFILE, or before any calls that allocate FDs when the FD table is almost full, just as the GC runs when we run out of memory. I wonder if this proposal could help: 'GC The Expensive Object Problem' http://www.eros-os.org/pipermail/e-lang/1999-May/002590.html Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
On approximately 6/16/2009 11:20 AM, came the following characters from the keyboard of Scott David Daniels: MRAB wrote: I was thinking along the lines of: def peek(self, size=None, block=True) If 'block' is True then return 'size' bytes, unless the end of the file/stream is reached; if 'block' is False then return up to 'size' bytes, without blocking I tend to prefer zero-ish defaults, how about: def peek(self, size=None, nonblocking=False) No, no, no! Double negatives are extremely easy to not code correctly. The lack of ease of not understanding of code containing double negatives quadruples, at least! Not so differently, I'm sure my sentences here are not easy to understand because I put the inverse logic in them in the places that are not the usual. -- Glenn -- http://nevcal.com/ === A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
Scott David Daniels Scott.Daniels at Acm.Org writes: MRAB wrote: I was thinking along the lines of: def peek(self, size=None, block=True) If 'block' is True then return 'size' bytes, unless the end of the file/stream is reached; if 'block' is False then return up to 'size' bytes, without blocking I tend to prefer zero-ish defaults, how about: def peek(self, size=None, nonblocking=False) Since blocking and non-blocking are already used to refer to different types of raw streams, another verb should be found for this option. Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL Certificate Validation
I have a few questions about validating SSL certificates. From what I gather, this validation occurs in the OpenSSL code called from _ssl.c. Is this correct? This question is really off-topic for python-dev. As a python-dev poster, you should do research upfront, and only post on what you consider facts. Where I'm going with this is I think all this checking needs to be part of certificate validation in the ssl module. If it isn't yet, I'd be happy to work on a patch for it. Please let me know what you think. I think you need to familiarize yourself much more with OpenSSL. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL Certificate Validation
On Tue, Jun 16, 2009 at 3:23 PM, Martin v. Löwismar...@v.loewis.de wrote: I have a few questions about validating SSL certificates. From what I gather, this validation occurs in the OpenSSL code called from _ssl.c. Is this correct? This question is really off-topic for python-dev. As a python-dev poster, you should do research upfront, and only post on what you consider facts. Martin, I told him to ask his question about _ssl internals on python-dev as he is new, and looking to work on some of the internals/make a patch for core. I didn't think that asking internals questions was a faux pas for the list, especially as he's looking to submit a patch to core. Where I'm going with this is I think all this checking needs to be part of certificate validation in the ssl module. If it isn't yet, I'd be happy to work on a patch for it. Please let me know what you think. I think you need to familiarize yourself much more with OpenSSL. I don't think that's called for, he is attempting to familiarize himself and simply inquiring about some of the internals. I'm sure he'll know plenty by the time the patch is more fully formed. -jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL Certificate Validation
This question is really off-topic for python-dev. As a python-dev poster, you should do research upfront, and only post on what you consider facts. Martin, I told him to ask his question about _ssl internals on python-dev as he is new, and looking to work on some of the internals/make a patch for core. I didn't think that asking internals questions was a faux pas for the list, especially as he's looking to submit a patch to core. Hmm. For somebody new to Python, I'm fairly skeptical that the SSL module is the best starting point. Where I'm going with this is I think all this checking needs to be part of certificate validation in the ssl module. If it isn't yet, I'd be happy to work on a patch for it. Please let me know what you think. I think you need to familiarize yourself much more with OpenSSL. I don't think that's called for, he is attempting to familiarize himself and simply inquiring about some of the internals. I'm sure he'll know plenty by the time the patch is more fully formed. But I really do believe that this is what he need to do next: familiarize himself with OpenSSL. There is a lot of APIs in that library, and it takes a while (i.e.: several months) to get productive, in particular since OpenSSL doesn't have the most intuitive API. From I want to know what features it currently has to I can contribute new features is really a looong way here. To give a little more guidance: find out what SSL_CTX_use_certificate_chain_file and SSL_CTX_set_verify do. Finding that out is really out of scope of python-dev, since it has nothing to do with Python. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL Certificate Validation
But I really do believe that this is what he need to do next: familiarize himself with OpenSSL. There is a lot of APIs in that library, and it takes a while (i.e.: several months) to get productive, in particular since OpenSSL doesn't have the most intuitive API. Well, I realized this as soon as I looked at the _ssl.c code... I was just hoping that someone would be able to give me a quick clarification on exactly what gets validated. If it's just the chain (which is what I suspect), I would like to submit a patch that does the rest of the validation (that a browser typically does: CN/hostname, NotBefore, NotAfter, etc.) in the ssl module. I was also hoping to find out what the consensus is about this: mainly, *should* that verification be done in the ssl module? Maybe this verification should somehow be done in OpenSSL, which would mean that I need to do a LOT more reading and go pester their mailing list instead. This is for issue 6273 ( http://bugs.python.org/issue6273 ). In your reply to that issue, it seemed to me like you were saying that these things were not getting checked in the ssl module (and, therefore, not in OpenSSL either): I find the patch incomplete, for formal and semantical reasons: a) it doesn't come with documentation or test suite changes, and b) it doesn't implement the typical certificate checks that browsers do, beyond validating that the certificate is valid - e.g. also validating that the certificate is issued to the host you are trying to connect to. I would like to do validation of server certificates in a project I'm working on, and I figured it would be better to be proactive and try to help create a patch than to just sit back and complain about it. It seems to me that this is a bug that you can't do peer certificate validation in httplib. If this isn't the place to ask these kinds of questions, I apologise. I can take the discussion elsewhere if I need to. Thanks, -Devin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL Certificate Validation
Devin Cook devin.c.c...@gmail.com wrote: Also, I have looked through the docs and code, but haven't been able to figure out exactly what is included in certificate validation. Is it just validating the chain? Does it check the NotBefore and NotAfter dates? I believe so, but you'll have to check the OpenSSL code. Does it check that the host the socket is connected to is the same as what's given in the CN field in the certificate? No. That, in general, doesn't work very well. The IETF working group on this is considering deprecating putting a hostname in the CN field at all, and just adding hostnames via the subjectAltName extension. The problem that's come up is that many computers don't have fixed IP addresses, and even with that the hostname is part of a different mapping of hostnames to IP addresses, which can also vary. I think that when the https: protocol scheme was written, it seemed like a good idea, but conventions on the Internet have changed a lot since then. Where I'm going with this is I think all this checking needs to be part of certificate validation in the ssl module. I don't think so. I put in hooks to let you do this in user code if you need to. See the archives for more discussion on this -- I'm not going to rehash it again. (This is really a question for OpenSSL mailing lists, or perhaps python-list.) Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL Certificate Validation
On Tue, Jun 16, 2009 at 5:31 PM, Devin Cookdevin.c.c...@gmail.com wrote: But I really do believe that this is what he need to do next: familiarize himself with OpenSSL. There is a lot of APIs in that library, and it takes a while (i.e.: several months) to get productive, in particular since OpenSSL doesn't have the most intuitive API. Well, I realized this as soon as I looked at the _ssl.c code... I was just hoping that someone would be able to give me a quick clarification on exactly what gets validated. If it's just the chain (which is what I suspect), I would like to submit a patch that does the rest of the validation (that a browser typically does: CN/hostname, NotBefore, NotAfter, etc.) in the ssl module. I was also hoping to find out what the consensus is about this: mainly, *should* that verification be done in the ssl module? Maybe this verification should somehow be done in OpenSSL, which would mean that I need to do a LOT more reading and go pester their mailing list instead. This is for issue 6273 ( http://bugs.python.org/issue6273 ). In your reply to that issue, it seemed to me like you were saying that these things were not getting checked in the ssl module (and, therefore, not in OpenSSL either): Also my initial bug report client-side cert support was a big fat typo on my part. face-palm'dly yours, jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL Certificate Validation
If this isn't the place to ask these kinds of questions, I apologise. I can take the discussion elsewhere if I need to. It really depends on what these questions are. If your question is I have this patch, is it correct?, then the question is entirely appropriate. If it is I just have barely looked at the API, can somebody please explain it all to me?, then this isn't appropriate for this list, and probably not appropriate elsewhere: anybody answering this question could just as well fix the original problem right away. So please do try to find the answer for yourself, with the (little) direction I gave. If you find that it takes a lot of effort, then you'll probably have to accept the bug as-is, and live with it. FWIW, I actually don't know the answer for sure, either, so I would have to research this myself, too. In any case, _ssl.c is *not* the place where any of the certificate validation actually happens - nor does it happen elsewhere in the Python source code, IIUC. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
Cameron Simpson wrote: I normally avoid non-blocking requirements by using threads, so that the thread gathering from the stream can block. If you have a thread dedicated to reading from that stream, then I don't see why you need to peek into the buffer. Just have it loop reading a packet at a time and put each completed packet in the queue. If several packets arrive at once, it'll loop around that many times before blocking. arguably read1() is a lousy name too, on the same basis. Certainly. My itch is that peek() _feels_ like it should be look into the buffer but actually can block and/or change the buffer. My problem with the idea of looking into the buffer is that it crosses levels of abstraction. A buffered stream is supposed to behave the same way as a deterministic non-buffered stream, with the buffer being an internal optimisation detail that doesn't exist as far as the outside world is concerned. Sometimes it's pragmatic to break the abstraction, but it should be made very obvious when you're doing that. So I'd prefer something like peek_buffer() to make it perfectly clear what's being done. Anything else such as peek() that doesn't explicitly mention the buffer should fit into the abstraction properly. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Functions that steal references (Re: [pygame] [patch] minor memory leaks...)
Lenard Lindstrom wrote: I assumed that since PyModule_AddObject is documented as stealing a reference, it always stole a reference. But in reality it only does so conditionally, when it succeeds. As an aside, is this a general feature of functions that steal references, or is PyModule_AddObject an oddity? I ask because I've been thinking about adding features to Pyrex for dealing with stolen references, and it could be important to know things like this. Also, if it's an oddity, it would be a good idea to mention this behaviour in the docs. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Functions that steal references (Re: [pygame] [patch] minor memory leaks...)
Benjamin Peterson schrieb: 2009/6/16 Greg Ewing greg.ew...@canterbury.ac.nz: Lenard Lindstrom wrote: I assumed that since PyModule_AddObject is documented as stealing a reference, it always stole a reference. But in reality it only does so conditionally, when it succeeds. As an aside, is this a general feature of functions that steal references, or is PyModule_AddObject an oddity? IIRC, It's an oddity. But it is a convenient oddity nonetheless. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
Greg Ewing greg.ewing at canterbury.ac.nz writes: Anything else such as peek() that doesn't explicitly mention the buffer should fit into the abstraction properly. peek() doesn't fit into the abstraction since it doesn't even exist on raw streams. While buffered and non-buffered streams have a reasonably similar API, expecting them to behave the same in all circumstances is IMO unrealistic. Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Functions that steal references (Re: [pygame] [patch] minor memory leaks...)
Christian Heimes wrote: But it is a convenient oddity nonetheless. What's convenient about it? Seems to me it's the opposite, since you can't just bail out if it fails, but have to decref the reference you thought it was going to take care of for you. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
On 17Jun2009 10:55, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Cameron Simpson wrote: I normally avoid non-blocking requirements by using threads, so that the thread gathering from the stream can block. If you have a thread dedicated to reading from that stream, then I don't see why you need to peek into the buffer. Just have it loop reading a packet at a time and put each completed packet in the queue. If several packets arrive at once, it'll loop around that many times before blocking. Yes, this is true. But people not using threads, or at any rate not dedicating a thread to the reading task, don't have such luxury. Are we disputing the utility of being able to ask how much might I read/peek without blocking? Or disputing the purpose of peek, which feels to me like it should/might ask that question, but doesn't. [...] My itch is that peek() _feels_ like it should be look into the buffer but actually can block and/or change the buffer. My problem with the idea of looking into the buffer is that it crosses levels of abstraction. A buffered stream is supposed to behave the same way as a deterministic non-buffered stream, with the buffer being an internal optimisation detail that doesn't exist as far as the outside world is concerned. Sometimes it's pragmatic to break the abstraction, but it should be made very obvious when you're doing that. So I'd prefer something like peek_buffer() to make it perfectly clear what's being done. Anything else such as peek() that doesn't explicitly mention the buffer should fit into the abstraction properly. It's perfectly possible, even reasonable, to avoid talking about the buffer at all. I'd be happy not to mention the buffer. For example, one can readily imagine the buffered stream being capable of querying its input raw stream if there's available now data. The raw stream can sometimes know if a read of a given size would block, or be able to ask what size read will not block. As a concrete example, the UNIX FIONREAD ioctl can generally query a file descriptor for instantly-available data (== in the OS buffer). I've also used UNIXen where your can fstat() a pipe and use the st_size field to test for available unread data in the pipe buffer. Raw streams which can't do that would return 0 (== can't guarentee any non-blocking data) unless the stream itself also had a buffer of its own and it wasn't empty. So I would _want_ the spec for available_data() (new lousy name) to talk about data available without blocking, allowing the implementation to use data in the IO buffer and/or to query the raw stream, etc. Cheers, -- Cameron Simpson c...@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/ For those who understand, NO explanation is needed, for those who don't understand, NO explanation will be given! - Davey D decos...@vnet.ibm.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] io.BufferedReader.peek() Behaviour in python3.1
On Sat, 13 Jun 2009 12:33:46 + (UTC) Antoine Pitrou solip...@pitrou.net wrote: This proposal looks reasonable to me. Please note that it's too late for 3.1 anyway - we're in release candidate phase. Once you have a patch, you can post it on the bug tracker. Thanks I will do that. Sometime in the next couple of weeks. Gratefully Frederick ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
