[issue1509060] Interrupt/kill threads w/exception
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1509060> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1572968] release GIL while doing I/O operations in the mmap module
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1572968> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1453973] addheaders for urlopen / open / xxxx_open
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1453973> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13451] sched.py: speedup cancel() method
Change by Josiah Carlson : -- nosy: -josiah.carlson, josiahcarlson ___ Python tracker <https://bugs.python.org/issue13451> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1043134] Add preferred extensions for MIME types
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1043134> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35913] asyncore: allow handling of half closed connections
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue35913> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13372] handle_close called twice in poll2
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue13372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6911] Document changes in asynchat
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue6911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4277] asynchat's handle_error inconsistency
Change by Josiah Carlson : -- nosy: -josiah.carlson, josiahcarlson ___ Python tracker <https://bugs.python.org/issue4277> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1442493] IDLE shell window gets very slow when displaying long lines
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1442493> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] add non-blocking read and write methods to subprocess.Popen
Josiah Carlson added the comment: Someone else can resurrect this concept and/ore patch if they care about this feature. Best of luck to future readers. -- stage: test needed -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1260171] subprocess: more general (non-buffering) communication
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1260171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] add non-blocking read and write methods to subprocess.Popen
Josiah Carlson added the comment: Non-blocking IO returning a None on no data currently available is new to me. It is new to me simply because I don't recall it ever happening in Python 2.x with any socket IO that I ever dealt with, and socket IO is my primary source for non-blocking IO experience. From my experience, reading from a socket would always return a string, unless the connection was closed/broken, at which point you got a bad file descriptor exception. Writing would return 0 if the buffer was full, or a bad file descriptor exception if the connection was closed/broken. The implementation of asyncore/asynchat at least until 3.0 shows this in practice, and I doubt this has changed in the intervening years since I updated those libraries for 3.0. My choice to implement this based on BrokenPipeError was based on reading Victor(haypo)'s comments from 2014-07-23 and 2014-07-24, in which he stated that in asyncio, when it comes to process management, .communicate() hides exceptions so that communication can finish, but direct reading and writing surface those exceptions. I was attempting to mimic the asyncio interface simply because it made sense to me coming from the socket world, and because asyncio is where people are likely to go if the non-blocking subprocess support in subprocess is insufficient for their needs. Note that I also initially resisted raising an exception in those methods, but Victor(haypo)'s comments changed my mind. Do you know a better method that would allow to distinguish between EOF (no future data, ever) and would block (future data possible) without calling process.poll()? Better is subjective when it comes to API, but different? Yes, it's already implemented in patch #8. BrokenPipeError exception when no more data is sendable/receivable. A 0 or b'' when writing to a full buffer, or reading from an empty buffer. This API tries to be the same as the asyncio.subprocess.Process() behavior when accessing stdin, stdout, and stderr directly. Returning None for non blocking I/O is standard in Python. In some contexts, yes. In others, no. The asyncio.StreamReader() coroutines .read(), .readline(), and .readexactly() return strings. Raw async or sync sockets don't return None on read. SSL-wrapped async or sync sockets don't return None on read. Asyncio low-level socket operations don't yield None on a (currently empty) read. In the context of the subprocess module as it exists in 3.4 (and 3.5 as it is unpatched), the only object that returns None on read when no data is available, but where data *might* be available in the future, is an underlying posix pipe (on posix platforms) - which isn't generally used directly. The purpose of this patch is to expose stdin, stdout, and stderr in a way that allows non-blocking reads and writes from the subprocess that also plays nicely with .communicate() as necessary. Directly exposing the pipes doesn't work due to API inconsistencies between Windows and posix, so we have to add a layer. I would argue that this layer of non-blocking (Windows and posix) pipe abstraction has much more in common with how asyncio deals with process IO, or how sockets deal with IO, than it does with pipes in general (the vast majority of which are blocking), so I would contend that it should have a similar API (no returning None). That said, considering that the expected use of non-blocking subprocess communication *does not* include using a multiplexer (select, poll, etc.), I have the sense that a few other higher-level methods might be warranted to ease use substantially, and which I would expect people would end up using much more often than the lower-level direct read/write operations (different names might make more sense here): .write_all(data, timeout=None) .read_available(bufsize=-1) (and .read_stderr_available) .read_exactly(bufsize, timeout=None) (and .read_stderr_exactly) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] add non-blocking read and write methods to subprocess.Popen
Josiah Carlson added the comment: I'm going to be honest; seeing None being returned from a pipe read feels *really* broken to me. When I get None returned from an IO read operation, my first instinct is there can't be anything else coming, why else would it return None? After changing writing to raise BrokenPipeError on a closed pipe, I've also changed reading to do the same and added a test to verify the behavior. -- Added file: http://bugs.python.org/file38691/subprocess_8.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] add non-blocking read and write methods to subprocess.Popen
Josiah Carlson added the comment: Okay, I'm sorry for falling asleep at the wheel on this. At this point, I don't expect this to go in for 3.5 - but if it has a chance, I'll do what I need to do to get it there. Let me know. I agree with the .communicate() not raising on broken pipe, and read/write_nonblocking raising on broken pipe. I'll get a patch in with the comments on the existing code review, along with those changes on Monday or Tuesday. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: First, with the use of Overlapped IO on Windows, BlockingIOError should never come up, and we don't even handle that exception. As for BrokenPipeError vs. EOF; while we do treat them more or less the same, we aren't killing the process or reporting that the process is dead, and we tell users to check the results of Popen.poll() in the docs. Could we bubble up BrokenPipeError? Sure. Does it make sense? In the case of Popen.communicate(), exposing the error and changing expected behavior doesn't make sense. I have implemented and would continue to lean towards continuing to hide BrokenPipeError on the additional API endpoints. Why? If you know that a non-blocking send or receive could result in BrokenPipeError, now you need to wrap all of your communication pieces in BrokenPipeError handlers. Does it give you more control? Yes. But the majority of consumers of this API (and most APIs in general) will experience failure and could lose critical information before they realize, Oh wait, I need to wrap all of these communication pieces in exception handlers. I would be open to adding either a keyword-only argument to the methods and/or an instance attribute to enable/disable raising of BrokenPipeErrors during the non-blocking calls if others think that this added level of control. I would lean towards an instance attribute that is defaulted to False. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: I submitted an issue to the tulip/asyncio bug tracker: https://code.google.com/p/tulip/issues/detail?id=170 And I am uploading a new patch that only includes non-tulip/asyncio related changes, as tulip/asyncio changes will eventually be propagated to Python. -- Added file: http://bugs.python.org/file35407/subprocess_7.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: First off, thank you everyone who has reviewed and commented so far. I very much appreciate your input and efforts. Does anyone have questions, comments, or concerns about the patch? If no one mentions anything in the next week or so, I'll ping the email thread. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Victor, I addressed the majority of your comments except for a couple stylistic pieces. Your annoyance with the short poll time for Windows made me re-read the docs from MS, which made me realize that my interpretation was wrong. It also made me confirm Richard Oudkerk's earlier note about ReadFile on overlapped IOs. I left similar notes next to your comments. On the method naming side of things, note that you can only write to stdin, and you can only read from stdout or stderr. This is documented behavior, so I believe the method names are already quite reasonable (and BDFL approved ;)). -- Added file: http://bugs.python.org/file34941/subprocess_6.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Richard: short timeouts are no longer an issue. Nothing to worry about :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: I ended up eliminating the overlapped IO cancel call on Windows. Better to be correct than to minimize internal state. Instead, we keep a reference to the overlapped IO object, and any attempts to write to the child stdin before the previous overlapped IO completes are kicked back as a 0 byte write. The communicate() method does the right thing with pre-existing non-blocking writes, whether input is passed or not. I also eliminated universal_newline support for non-blocking reads and writes to prevent error conditions on edge cases: On the write side of things, you could end up with a partial multi-byte character write, which with universal newlines, would be impossible to finish the send using the public API without first modifying state attributes on the Popen object (disabling universal newlines for a subsequent bytes write_nonblocking() call). On the read side of things, if you get a partial read of a multi-byte character, then the subsequent decode operation would fail with a UnicodeDecodeError. Though you could pull the original bytes from the exception, that's an awful API to create. -- Added file: http://bugs.python.org/file34851/subprocess_4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Changes by Josiah Carlson josiah.carl...@gmail.com: Added file: http://bugs.python.org/file34861/subprocess_5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: No, the problem is that that ov.cancel() will attempt to cancel the IO, but a subsequent ov.getresult(True) doesn't always return what was *actually* written to the pipe unless you explicitly wait for the result to be available. But even if you explicitly wait for ov.pending to be False, even if you alternate between checking for ov.pending to be False and for WaitForSingleObject() to return that the event was signaled (which may never actually happen, according to my tests), ov.getresult(True) could *still* raise an exception (that same WinError 995), and we *still* don't know how much data was sent. As one of your comments on subprocess_2.patch, you said: Should probably add an assertion that 512 bytes was written. That's not always the case. I found several odd byte writes, and some writes that were whatever blocksize I'd chosen minus 32 bytes (though not reported on the write side due to the aforementioned exception/counting error, but the child process read an odd number of bytes). How about you take a look at the patch I'm uploading now. It gets rid of the loop in write_nonblocking(), as per Giampaolo's request adds a blocksize parameter on reading, and because I was in there, I added a timeout to writing. If in this new patch I'm doing something wrong, and you can explain via code how to guarantee that ProcessTestCase._test_multiple_passes doesn't fail, I'd really appreciate it. -- Added file: http://bugs.python.org/file34794/subprocess_3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: I added the chunking for Windows because in manual testing before finishing the patch, I found that large sends on Windows without actually waiting for the result can periodically result in zero data sent, despite a child process that wants to read. Looking a bit more, this zero result is caused by ov.cancel() followed by ov.getresult() raising an OSError, specifically: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request That causes us to observe on the Python side of things that zero data was sent for some writes, but when looking at what the child process actually received, we discover that some data was actually sent. How much compared to what we thought we sent? That depends. I observed in testing today that the client could receive ~3.5 megs when we thought we had sent ~3 megs. To make a long story short-ish, using Overlapped IO with WriteFile() and Overlapped.cancel(), without pausing between attempts with either a sleep or something else results in a difference in what we think vs. reality roughly 87% of the time with 512 byte chunks (87 trials out of 100), and roughly 100% of the time with 4096 byte chunks (100 trials out of 100). Note that this is when constantly trying to write data to the pipe. (each trial is as many Popen.write_nonblocking() calls as can complete in .25 seconds) Inducing a 1 ms sleep between each overlapped.WriteFile() attempt drops the error rate to 0/100 trials and 1/100 trials for 512 byte and 4096 byte writes, respectively. Testing for larger block sizes suggests that 2048 bytes is the largest send that we can push through and actually get correct results. So, according to my tests, there isn't a method by which we can both cancel an overlapped IO while at the same time guaranteeing that we will account exactly for the data that was actually sent without adding an implicit or explicit delay. Which makes sense as we are basically trying to interrupt another process in their attempts to read data that we said they could read, but doing so via a kernel call that interrupts another kernel call that is doing chunk-by-chunk copies from our write buffer (maybe to some kernel memory then) to their read buffer. Anyway, by cutting down what we attempt to send at any one time, and forcing delays between attempted sends, we can come pretty close to guaranteeing that child processes don't have any sends that we can't account for. I'll try to get a patch out this weekend that encompasses these ideas with a new test that demonstrates the issue on Windows (for those who want to verify my results). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Submitting an updated patch addressing Giampaolo's comments. -- Added file: http://bugs.python.org/file34774/subprocess_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Should have uploaded this yesterday, but I got caught up with typical weekend activities. The docs probably need more, and I hear that select.select() is disliked, so that will probably need to be changed too. -- Added file: http://bugs.python.org/file34743/subprocess.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: All of the standard tests plus another few that I added all pass on Windows and Linux. I've got some cleanup and a couple more tests to add tomorrow, then I'll post a patch. I ended up not using any overlapped IO cancellation in the Windows variant of communicate(), as there is no benefit to doing so, and the function is as re-entrant as the original. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Quick update before I head to bed. Thank you for the input, I had gotten the individual async calls working a couple days ago, and I was just working to replace the communicate() method for Windows. Yes, I'm using asyncio._overlapped, though asyncio uses subprocessing, so I needed to defer import of _overlapped until one of a couple crucial methods were being called in order to prevent issues relating to circular imports. I also ended up moving asyncio.windows_utils.pipe out to subprocess, as copying/pasting it for a third 'create an overlapped-io pipe' implementation for the standard library just doesn't make a lot of sense, and another circular import seemed like a bad idea. If/when subprocess goes away completely, someone else can move the function back. With overlapped IOs able to be cancelled, it's not all that bad to make a completely re-entrant communicate() without threads, though I made a few mistakes in my first pass tonight that I need to fix tomorrow. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Had some time to work on this today. I was missing something in my earlier versions of the code, and have managed to get overlapped IOs to work, so at least I'm not quite so far behind the dozen or so core developers who know more about the Windows pieces than I do. Richard, thank you for the post, I wasn't looking hard enough for how to get overlapped IOs to work, and your message made me look harder. On Linux, it is trivial to support the blocking communicate() and non-blocking additions. There's only one weirdness, and that's the fcntl bit flipping during write. On Windows, it's not all that difficult to switch to using overlapped IOs for *all* writes, and maybe even for communicate()-based reads, which would remove the need for threads. Ironically enough, non-blocking reads actually *don't* require overlapped IO thanks to PeekNamedPipe, which could even be used to cut the number of extra threads from 2 to 1 in communicate(). Now that I've got it working, I can do one of the following (from least changes to the most): 1. Add a nonblocking flag, which pre-flips the fcntl bit in Linux and uses overlapped IO on writes in Windows - this would be documented to remove the ability to call communicate() 2. As an alternative to #1, I can create a new class that lacks the communicate() method and adds the non-blocking methods 3. Gut/rewrite the Windows-based communicate() function to use overlapped IO on everything, removing the threads, and making it at least superficially like Linux (prepare your overlapped IO, then use WaitForMultipleObjects() to multiplex), while also adding the non-blocking methods Unless someone brings up an important counterpoint, I'll work on #3 tonight or tomorrow evening to get an initial code/test patch, with docs coming shortly after (if not immediately). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Due to some rumblings over on the mentors list and python-dev, this is now getting some love. Guido has stated that something should make it into the subprocess module for 3.5 in this email: https://groups.google.com/d/msg/dev-python/I6adJLIjNHk/Usrvxe_PVJIJ His suggested API is: proc.write_nonblocking(data) data = proc.read_nonblocking() data = proc.read_stderr_nonblocking() I've implemented the API for Windows and Linux, and below are my findings. On the Linux side of things, everything seems to be working more or less as expected, though in writing tests I did need to add an optional argument to qcat.py in order to have it flush its stdout to be able to read from the parent process. Odd, but whatever. Also, Richard Oudkerk's claims above about not needing to use fcntl to swap flags is not correct. It's necessary to not block on reading, even if select is used. Select just guarantees that there is at least 1 byte or a closed handle, not that your full read will complete. On the Windows side of things, my assumptions about WriteFile() were flawed, and it seems that the only way to make it actually not block if/when the outgoing buffer is full is to create a secondary thread to handle the writing*. I've scoured the MSDN docs and there doesn't seem to be an available API for interrogating the pipe to determine whether the buffer is full, how full it is, or whether the write that you'd like to do will succeed or block. * This applies even with the use of asyncio. Because the intent for the call is to return more or less immediately, a thread still has to be created to handle the event loop for IO completion, which means that asyncio can't prevent the use of threads and not block without basically integrating an event loop into the caller. Considering that the Windows communicate() method already uses threads to handle reading and writing, I don't believe it is a big deal to add it for this situation too. Any major objections? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Changes by Josiah Carlson josiah.carl...@gmail.com: -- versions: +Python 3.5 -Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11265] asyncore does not check for EAGAIN and EPIPE errno
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Giampaolo pinged me over email... These additional conditions look good, and should be targeted for 3.3 . Thank you :) -- nosy: +josiahcarlson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11265 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8684] improvements to sched.py
New submission from Josiah Carlson josiahcarl...@users.sourceforge.net: This patch is against Python trunk, but it could be easily targeted for Python 3.2 . It is meant to extract the scheduler updates from issue1641 without mucking with asyncore. It's reach is reduced and simplified, which should make maintenance a bit easier. -- assignee: giampaolo.rodola components: Library (Lib) files: sched.patch keywords: needs review, patch, patch messages: 105483 nosy: giampaolo.rodola, josiahcarlson priority: low severity: normal status: open title: improvements to sched.py type: feature request Added file: http://bugs.python.org/file17289/sched.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8684 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Some prodding from Giampaolo got me to pull out and simplify the sched.py changes here: issue8684 . That should be sufficient to add scheduling behavior into async socket servers or otherwise. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8543] asynchat documentation issues
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: The suggested documentation changes sound good to me. Those items that aren't documented may need a note that they are deprecated and will be removed in the future, but I'd consider that optional. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8543 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1023290] proposed struct module format code addition
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I'm not a big fan of the names, but as long as the functionality exists, people can easily alias them as necessary. I've not actually looked at the patch, but as long as it does what it says it does, it looks good. My only question, does it makes sense to backport this to trunk so we get this in 2.7? I personally would like to see it there, and would even be ok with a limitation that it only exists as part of longs. If someone has the time to backport it; cool. If not, I understand, and could live with it only being in 3.x . Thank you for taking the time and making the effort in getting this into a recent Python :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1023290 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: handle_expt_event was removed in the test classes because it is no longer being used by any of the tests. None of them send OOB data (also known as priority data), so handle_expt_event should never be called. When I have a chance to compare your patch to mine (Monday, likely), I'll comment then. In terms of handle_expt_event should handle the low level expt event called from select, what you don't seem understand is that and expt event is not an exception event, it's a we got OOB data event, and with the patches, it is called at *exactly* the time it should be: when there is OOB data. It used to be mistakenly called in the select loop whenever any sort of non-normal condition happened on the socket, which was a serious design flaw, and lead to significant misunderstanding about the purpose of the method. With the _exception() call as-is, it now behaves like the asyncore.poll2() function, which is the right thing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6550 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Originally, handle_expt_event() was described as handles OOB data or exceptions, but over-using handle_expt_event() as an error/close handler is a bad idea. The function asyncore.readwrite() (called by asyncore.poll2()) does the right thing WRT handle_expt_event(), which it makes sense to apply to the standard select-based asyncore.poll(). That's what this does (in addition to fixing the close case that you pointed out). In terms of only implementing low-level stuff, this is still the case. You still only need to implement handle_*(), not handle_*_event() . But now, handle_expt_event() isn't written to do more than it should have been doing in the first place. I've updated the patch to include semantics for actually handling OOB data, which I've verified by using a slightly modified pyftpdlib (remove the socket option calls to set socket.SO_OOBINLINE) and it's tests on both Windows and Ubuntu 8.04 (I also ran the full Python test suite on my Ubuntu install, and any failures were obviously not asyncore/asynchat related). -- Added file: http://bugs.python.org/file14596/asyncore_fix_refused-3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6550 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: Removed file: http://bugs.python.org/file14581/asyncore_fix_refused.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6550 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: Removed file: http://bugs.python.org/file14585/asyncore_fix_refused-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6550 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Firstly, it expects that handle_expt_event() is for handling exceptional conditions. This is not the case. handle_expt_event() is meant for handling OOB or priority data coming across a socket. FTP and some other protocols use this. I forgot to fix it earlier, which is why it's making it into this patch. Secondly, I pulled the part that was inside handle_expt_event() that was being used to find the exception and pulls it out into _exception(), removing the previous behavior (wrt to the broken API), and replacing it with something that is cleaner and more correct (assuming sockets). To respond to it being an issue that the object has a socket with a getsockopt(), I can fix it to handle the AttributeError case. Would you be willing to try this out given my explanation as to why I changed your patch? -- Added file: http://bugs.python.org/file14585/asyncore_fix_refused-2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6550 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: The attached patch cleans up the remnants of the handle_expt is for exceptions, which isn't the case, as well as makes the connection refused fix actually work on Windows. Nirs, could you verify this on *nix? -- assignee: - josiahcarlson keywords: +needs review Added file: http://bugs.python.org/file14581/asyncore_fix_refused.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6550 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1314572] Trailing slash redirection for SimpleHTTPServer
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: The other patch is more correct. Closing. -- resolution: - duplicate status: open - closed superseder: - SimpleHTTPServer directory-indexing bug fix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1314572 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I installed 3.1rc1 on my OS X (10.5.?) machine, updated asynchat, and ran the test with and without my change. Without my change, it breaks in the way described numerous times. With my change, it seems to work fine on OS X, linux, and Windows for me. Looking at line 106 in py3k/Lib/asyncore.py (http://svn.python.org/view/python/branches/py3k/Lib/asyncore.py? annotate=73183) gets me obj.handle_close(). Your tracebacks show line 106 calling obj.handle_expt_event(), which is the old code. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: If it's failing, and asyncore is still in 3.1, then I would argue yes. I'll submit a fix to trunk and 3.1 based on my version below (unless anyone has any outstanding concerns, or believes that it doesn't work for them). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: Removed file: http://bugs.python.org/file13934/asyncore_fix_mac_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Fixed in trunk in 73182, fixed in py3k in 73183. Closing as fixed. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1563] asyncore and asynchat incompatible with Py3k str and bytes
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: You can probably close this unless someone says otherwise. asyncore/asynchat work in Python 3.0+, as long as only bytes are passed. As of right now, this is a request for documentation stating you can only send/receive bytes...which may be implicit with the fact that we are reading/writing to sockets. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1563 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: One of the issues with using the method that Giampaolo describes, which I explained to him, is that generally if someone sends you data, you want to receive it. You can get both data and the signal that someone disconnected, in particular, with asynchat's .close_when_done() method. If you get a read signal, generally you want to try to pull more data before you close, because when you close, you can never get any pending data again. For example, if I were to use: chat.send(commit\n) chat.close_when_done() It's very likely that on even moderate latency network connections (10ms), the other end would get the close signal at the same time as the read for the commit\n message. But since they close before they read, they never get the commit message, and maybe a transaction is reverted. My primary concern is keeping the library correct. Adding attributes in the test cases don't bother me. Jean: on what platform are you experiencing the hang? Are you using trunk or 3.1? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Ok, so I was running test_asyncore and not test_asynchat. The issue on OS X is that when a new socket is connecting, select.poll() shows the socket as being writable when it first connects, but using my variant, .connected is False while it's waiting for the connection. Options: If we swap the pollin/pollout conditions, that would fix the hang on mac, but cause the hang on others (I'm not really worried about not doing a write after a read, .push() in asynchat performs an immediate send). If we get rid of the .connected check, then when a .read() fails and causes a close, then we get an exception in handle_write(). If we swap to Giampaolo's variant, then we could lose data that was legitimately sent by a client. I've got a version that doesn't hang on OS X, and should work on all platforms. It's ugly, but it works. -- Added file: http://bugs.python.org/file13934/asyncore_fix_mac_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: Removed file: http://bugs.python.org/file13915/asyncore_fix_mac.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: Removed file: http://bugs.python.org/file13918/unnamed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Here's an option that doesn't use .connected, which some people have expressed distaste for. def readwrite(obj, flags): try: if flags select.POLLIN: obj.handle_read_event() if flags select.POLLOUT: obj.handle_write_event() if flags select.POLLPRI: obj.handle_expt_event() if flags (select.POLLHUP | select.POLLERR | select.POLLNVAL): obj.handle_close() except socket.error, e: if e.args[0] not in (EBADF, ECONNRESET, ENOTCONN, ESHUTDOWN, ECONNABORTED): obj.handle_error() else: obj.handle_close() except _reraised_exceptions: raise except: obj.handle_error() It works on OS X and should work just as well on other platforms. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Looking at trunk, it seems like one reasonable option is to swap the order of handle_close() and handle_expt_event() testing and calls. That would keep all reading/writing before handle_close(), which should be correct. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: It would seem that we need to be more defensive in our calls. We need to check to make sure that the socket isn't closed before calling read/write/expt events. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Mark, try this: if flags select.POLLIN and (obj.connected or obj.accepting): obj.handle_read_event() if flags select.POLLOUT and obj.connected: obj.handle_write_event() if flags select.POLLPRI and obj.connected: obj.handle_expt_event() if flags (select.POLLHUP | select.POLLERR | select.POLLNVAL): obj.handle_close() -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I went ahead and plugged my mac in (which reminded me of why I unplugged it in the first place), and I'm able to reproduce your error in the test after my proposed modifications. One thing to remember is that the test is broken; we rely on a .connected attribute, which the test objects lack. The attached patch fixes the test on my OS X machine, which you should test. -- keywords: +patch Added file: http://bugs.python.org/file13915/asyncore_fix_mac.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5798] test_asynchat fails on Mac OSX
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: As an aside, I was testing against trunk, not 3.1b1 . -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5798 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I'm not defending the documentation, I'm merely reposting it. The documentation for asyncore says, The full set of methods that can be overridden in your subclass follows: The documentation for asynchat says, To make practical use of the code you must subclass async_chat, providing meaningful collect_incoming_data() and found_terminator() methods. The asyncore.dispatcher methods can be used, although not all make sense in a message/response context. How can we make the documentation better? I'm too close to the documentation to really know how to improve it. Ideas? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: IIRC, there was a threat to remove asyncore because there were no maintainers, no one was fixing bugs, no one was improving it, and no one was really using it (I believe the claim was that people were just using Twisted). The patches that were ultimately committed to 2.6 and 3.0 were committed 3 months prior to 2.6 release, after having languished for over a year because no one would review them. If people care about where asyncore/asynchat are going, then it is *their* responsibility to at least make an effort in paying attention at least once every few months or so. The delayed calls feature discussed in the current bug doesn't alter the behavior of previously existing code, except there are additional checks for pending tasks to be executed. If people never use the call_later() API, it is unlikely they will experience any difference in behavior. If you are concerned about the sched module, I'd be happy to do a search for it's use to verify that people aren't relying on it's internal implementation, only the features of it's external API. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I'm happy to let them know proposed changes now that I know issues exist, but you have to admit that they were pretty under-the-radar until 4-5 months *after* 2.6 was released. If there is a mailing address that I can send proposed changes to asyncore so that they can have a say, I'd be happy to talk to them. Generally, what you are saying is that I'm damned if I do, damned if I don't. By taking ownership and attempting to fix and improve the module for 2.6 (because there were a bunch of outstanding issues), people are pissed (generally those who use Zope and/or medusa). Despite this, other people have continued to use it, and have been pushing for new features; event scheduling being one of the major parts. Pulling asyncore out of Python is silly. Not improving the module because of fear of breakage is silly. I'm happy to hear suggestions from the Zope crew, but I'm only going to put as much effort in communicating with them as they do me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Here's a question: How do we fix 2.6? From what I've read, the only answer I've heard is revert to 2.5 in 2.6.2, which has the same issues as adding True/False in 2.2 . I agree that Zope not working in 2.6 is a problem, I agree that the documentation for asyncore is lacking, I agree that I probably wasn't as vocal as I could have been prior to the changes, I agree that 3rd parties relying on internal implementation details not covered in the limited documentation is a problem, I agree that we need to figure something out for asyncore 2.7 and beyond, I agree that we need to figure something out for asyncore 2.6 issues related to Zope and Medusa, ... I'm happy to take the blame for changing asyncore internals in Python 2.6 . And I've not stated otherwise in any forum. At the time I thought it was the right thing to do. If I could change the code retroactively, I would probably do so. But it seems to me that fork asyncore, pull asyncore out of the stdlib, and revert to 2.5 are all variants of the cliche throwing the baby out with the bathwater. There are good bug fixes in 2.6, and depending on how much of the internals that Zope and/or medusa rely on, we might even be able to write a short wrapper/adapter to throw in to Zope and/or asyncore. I'll contact Tres and Jim, and hopefully be able to come to some reasonable solution. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: To be wholly clear about the issues, it's not with asyncore, the core asynchronous library, it's with asynchat and the internal changes to that. Any changes to asyncore were to fix corner cases and exceptions. No API, internal or external was changed. People who subclassed from asyncore should have no problems. People who subclassed from asynchat may have problems. If we want to revert selected changes to asynchat, that's fine with me. AFAICT, there is only 1 substantial bugfix in asynchat (if your text terminator isn't discovered in the first ac_in_buffer_size bytes read since the last terminator, your connection will hang), which is easily pulled out. Offering a compatibility mode is also relatively easy. Six months ago you were 'eh' with what was going on with the asyncore libraries (see messages from early October). Over a year ago everyone on python-dev cared so little about the libraries that it was preferred to give me commit access than for someone to review the code. Now everyone seems willing and happy to remove the library because it is unsalvageable. Ultimately the change that broke Zope/medusa was replacing the use of asynchat.fifo with a deque, and getting rid of ac_out_buffer. Those are *tiny* changes that we can change back, temporarily pull into Zope, and tweak Medusa to fix (I'd be happy to offer a patch to AMK to produce Medusa 0.5.5). As for your subclassing is bad comment, Twisted, wxPython, SocketServer (SimpleXMLRPCServer, TCPServer, ...), sgmllib.SGMLParser, etc., all use subclassing as part of their APIs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5661] asyncore should catch EPIPE while sending() and receiving()
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I think that catching one more potential failure modes is reasonable. I'm probably going to apply a variant of this patch to pull the sequence into a frozenset for quick lookups, and so that we don't need to keep updating two different places if/when new failure conditions arise. -- assignee: - josiahcarlson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5661 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Your analysis WRT handle_expt_event() is correct. I've been meaning to fix that for a while, but I forgot to do it in 2.6/3.0 with all of the other changes/fixes. Looking at the docs, you are also right about POLLNVAL. I also don't *know* what to do when presented with POLLERR, but few socket errors are transient (transient errors should be handled by the underlying stacks), so I agree with you that they should just be closed. I went ahead and made the changes as you have suggested, and also made the same change with the select-based loop. Errors on the socket just result in closing the socket, resulting in _exception() - close(). Thinking about it, I've also had a change of heart, and added a frozenset object called 'ignore_log_types', which specifies the log types to ignore. By default it is populated with 'warning', which squelches all currently existing unhandled * event bits. If you use self.log(arg) or self.log_info(one_arg), those lines are unchanged. Handle_error() uses the error type, which is also nice, but which you can also suppress with ease (though you really should be logging them so you can fix your code). I've attached a version that I think is pretty reasonable. Comments? Questions? -- keywords: +needs review, patch versions: +Python 2.7 -Python 2.6 Added file: http://bugs.python.org/file13516/async_no_warn.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1161031 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2073] asynchat push always sends 512 bytes (ignoring ac_out_buffer_size)
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: When push is called in the current trunk (as of 2.6), the data is automatically chopped up into self.ac_out_buffer_size blocks for later writing. In order to force the use of the asynchat.simple_producer class (which takes an argument for the block size to send), you must pass the producer itself with push_with_producer() . Closing as out of date. -- resolution: - out of date status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2073 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: You are right. Handling OOB data is within the exceptional condition that the select document specifies. I've added a check for error conditions within handle_expt_event(), which induces a handle_close() on discovery of an error, handle_expt() otherwise. One thing to consider is that when there is OOB data, and when handle_expt() isn't overridden, we will get churn because that data will never be read from the socket. I'm thinking about tossing a handle_close() as part of the default handle_expt() call. Attached is an updated patch without the handle_close() in handle_expt(). -- Added file: http://bugs.python.org/file13517/async_no_warn.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1161031 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: Removed file: http://bugs.python.org/file13516/async_no_warn.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1161031 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1370380] async_chat.push() can trigger handle_error(). undocumented.
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: -- resolution: - wont fix status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1370380 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Fixed the close() call and committed to trunk. Python 2.6 tests pass with the new version of the library. Calling it good :) . -- keywords: -needs review resolution: - accepted status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1161031 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I fixed some bugs with my patch, merged in Giampaolo's tests and documentation, and altered the API to match Giampaolo's API almost completely. This new version differs from Giampaolo's patch only in underlying implementation; this uses a modified sched.py, and doesn't have a standard execute outstanding methods function built into asyncore (asynchat.scheduled_tasks.run(time.time()) is sufficient). The major difference is that the modifications to sched.py offer a fast cancel/reschedule operation, which Giampaolo's lacks. -- Added file: http://bugs.python.org/file13524/scheduler.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: Removed file: http://bugs.python.org/file13238/scheduler_partial.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2073] asynchat push always sends 512 bytes (ignoring ac_out_buffer_size)
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: The spare 512 values are for code that I expect no one is actually using. In terms of increasing the buffer size from 4096 to something larger, that can be done, but I think that more than just a 10mbit switched lan test should be performed. I would tend to believe that a larger size would tend to perform better, but my concern is sending blocks too large to the OS and having it say sorry, but you need to chop it up more via sock.send() returning 1 or 2 (I've experienced this on Windows with stdio and sockets). Considering how easy it is to adjust, and considering that the value is respected everywhere that matters (performance-conscious users aren't using simple_producer, nor are they using dispatcher_with_send), I think that 4096 is sufficient for the time being. Adjusting it up to 16k in Python 2.8/3.2 seems reasonable to me. Someone ping me in a year ;) . -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2073 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue909005] asyncore fixes and improvements
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Just to make this clear, Aleksi is proposing close() should be called automatically by some higher-level functionality whether a user has overridden handle_close() or not. With the updated asyncore warning suppression stuff, overriding handle_close() for the sake of suppressing the warnings should no longer be necessary. While I can see that it would be *convenient* if close() was automatically called, the method is called handle_close(), and there is an expectation about the implementation thereof. For example, you call socket.recv() in handle_read(), you call socket.send() in handle_write(), call socket.accept() in handle_accept(). Is it too much to expect that a user will call .close() inside handle_close()? The answer to that last question is a no, btw. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue909005 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I don't believe this should be closed. The functionality is still desired by me and others who have posted on and off since the patch was created. This patch definitely needs some love (tests mostly). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1191964 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5406] asyncore doc issue
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Actually, that's exactly what it does. If the count is missing, it defaults to None. The code that is executed is exactly: if count is None: while map: poll_fun(timeout, map) It will loop until the map is empty. -- message_count: 1.0 - 2.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5406 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5406] asyncore doc issue
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Well...the loop can also die if an uncaptured exception is raised, but I'm not sure that is necessary to spell out explicitly. -- message_count: 2.0 - 3.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5406 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Forest: To answer your question, yes, that blog post discusses a better variant of sched.py , but no, there isn't a bug. I should probably post it some time soon for 2.7/3.1 inclusion. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: I've just attached a patch to sched.py and asyncore.py to offer a richer set of features for sched.py, with a call_later() function and minimal related classes for asyncore.py to handle most reasonable use-cases. There is no documentation or tests, but I can add those based on Giampaolo's tests and docs if we like this approach better. Added file: http://bugs.python.org/file13237/scheduler_partial.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Changes by Josiah Carlson josiahcarl...@users.sourceforge.net: Removed file: http://bugs.python.org/file13237/scheduler_partial.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Here's a better patch without tabs. Added file: http://bugs.python.org/file13238/scheduler_partial.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue777588] asyncore is broken for windows if connection is refused
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: According to Garth, sockets that don't connect on Windows get put into the error sockets list. According to John, you need to poll sockets to determine whether or not the attempted connection was refused. If Garth is right, the problem is fixed, though we aren't quite retrieving the correct error code on win32. If John is right, we need to repeatedly check for error conditions on sockets that are trying to connect to a remote host, and the problem is not fixed. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue777588 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Considering that we're looking at 2.7 and 3.1, I think that (paraphrased) logging fallout from changes to 2.4 are a bit out-of- date. People who have continued to use asyncore/asynchat in the last 4 years have already changed their code. People who write new code already deal with the logging. Does anyone have a compelling reason as to why we shouldn't just close this bug? ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1161031 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5097] asyncore.dispatcher_with_send undocumented
Josiah Carlson josiahcarl...@users.sourceforge.net added the comment: Feel like writing some documentation? ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5097 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4332] asyncore.file_dispatcher does not use dup()'ed fd
Josiah Carlson [EMAIL PROTECTED] added the comment: Oy. You are right. Fixed in Py3k in r67286, in trunk (2.7) in r67287, and 2.6-maintenance in r67288. -- resolution: - fixed status: open - closed ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4332 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3890] ssl.SSLSocket.recv() implementation may not work with non-blocking sockets
Josiah Carlson [EMAIL PROTECTED] added the comment: I agree with Giampaolo. In the case of non-blocking sockets, if reading from the ssl stream fails because there is no data on the socket, then sitting in a while loop is just going to busy-wait until data is discovered. Never mind that the reference to sendall should be replaced by recv. Whether to 'continue' or 'raise' should be determined by whether the socket is blocking. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3890 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4078] asyncore fixes are not backwards compatible
Josiah Carlson [EMAIL PROTECTED] added the comment: Zope's medusa was relying on internal details of asyncore (the ac_out_buffer attribute), which is no longer applicable. It also seems as though much of medusa itself borrows from asynchat.async_chat, which suggests that it should subclass there. -- resolution: - wont fix ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4078 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1541] Bad OOB data management when using asyncore with select.poll()
Josiah Carlson [EMAIL PROTECTED] added the comment: While handle_expt() is documented has only being called when OOB data is known, it has become the catch-all for something strange is happening on the socket. The reason it wasn't changed/fixed in Python 2.6 is because a new method would need to be added, which would break previously existing asyncore- derived applications. I might still be able to fix it for 3.0 . -- assignee: akuchling - josiahcarlson nosy: +josiahcarlson ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1541 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1541] Bad OOB data management when using asyncore with select.poll()
Josiah Carlson [EMAIL PROTECTED] added the comment: 3.0 is a no-go, no non-documentation changes allowed. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1541 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1745035] DoS smtpd vulnerability
Josiah Carlson [EMAIL PROTECTED] added the comment: The patch does not work as Giampaolo intends. If the patch were applied as-is, no emails longer than 998 bytes could be sent. Instead, incrementing linelen in the collect_incoming_data() method should only be performed if self.terminator == '\r\n'. I can apply a modified version of this patch against trunk after 2.6 is released. Backports to 2.5 and 2.6 should then be discussed. -- assignee: barry - josiahcarlson nosy: +josiahcarlson ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1745035 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson [EMAIL PROTECTED] added the comment: Thank you for the report (fixed in the newly attached version) :) . Added file: http://bugs.python.org/file11602/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3904] asynchat async_chat __init__() arguments changed in python 2.6
Josiah Carlson [EMAIL PROTECTED] added the comment: Fixed documentation in revision 66510. Also, the parameters changed long before rc2. ;) -- resolution: - fixed status: open - closed ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3904 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson [EMAIL PROTECTED] added the comment: I have an updated sched.py module which significantly improves the performance of the cancel() operation on scheduled events (amortized O(log(n)), as opposed to O(n) as it is currently). This is sufficient to make sched.py into the equivalent of a pair heap. From there, it's all a matter of desired API and features. My opinion on the matter: it would be very nice to have the asyncore loop handle all of the scheduled events internally. However, being able to schedule and reschedule events is a generally useful feature, and inserting the complete functionality into asyncore would unnecessarily hide the feature and make it less likely to be used by the Python community. In asyncore, I believe that it would be sufficient to offer the ability to call a function within asyncore.loop() before the asyncore.poll() call, whose result (if it is a number greater than zero, but less than the normal timeout) is the timeout passed to asyncore.poll(). Obviously the function scheduler would be written with this asyncore API in mind. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1641 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3899] test_ssl.py doesn't properly test ssl integration with asyncore
Josiah Carlson [EMAIL PROTECTED] added the comment: Being able to test the async features of both sides of the SSL connection is a good thing. Also, the subclass provides a useful example for users who want to use asyncore and ssl servers without blocking on an incoming connection. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3899 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Changes by Josiah Carlson [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11412/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson [EMAIL PROTECTED] added the comment: I like Skip's version better, because it's closer to the dbm specification instead of trying to mimic bsddb (first, last, etc.). I'd like to keep such things out. dbm.sqlite is meant as a potential replacement of dbm.bsddb. Since people do use the extra methods (.first(), .last(), etc.), not having them could lead to breakage. Separating them out into a subclass (regular open doesn't have it, but btopen does), along with all of the other order guarantees (the ORDER BY clauses in the SQL statements), could keep it fast for people who don't care about ordering, and keep it consistent for those who do care about ordering. Attached you will find an updated version. Added file: http://bugs.python.org/file11467/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson [EMAIL PROTECTED] added the comment: I would find it strange to potentially ruin performance just for a guarantee which has no useful purpose. Benchmarks to prove or disprove performance changes? Subclasses to offer different order by semantics (see the version I uploaded for an example)? Consistent behavior wrt dictionaries? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson [EMAIL PROTECTED] added the comment: If you like, but ordering semantics is something which is just as easily done in Python, so I don't understand the point of integrating it in the dbm layer... Actually, the db layer is *exactly* where it should be implemented, especially when an index can already exist (as is the case with the implementation I provided), especially when the total size of keys can easily overflow memory. Implementing ordering semantics in Python would be a waste of what the db is good at: storing and ordering data. It sounds like an example of foolish consistency to me. The performance characteristics are certainly too different to consider dbm.anything a transparent replacement for standard dicts. And dbm.sqlite only accepts strings, not the wide range of datatypes that dicts accept as keys and values... so, given the big picture, I don't see why you care about such a mostly pointless detail. No one here has ever claimed that dbm should be a replacement for dicts, even if shelve attempts to do so. This module should provide a shelve interface that mirrors bsddb's interface. One of the things that was offered earlier was that since sqlite can store ints, floats, etc., as keys, maybe we should offer that ability. I think that the base should act like a regular dbm object. I think that a key-ordering should be available. And if we are to offer arbitrary keys (ints, floats, unicode strings, byte strings, or None), it should be unordered like the base version. I've uploaded a new copy of sq_dict that offers unordered shelve and arbitrary keys in a shelve db. Added file: http://bugs.python.org/file11470/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Changes by Josiah Carlson [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file11467/sq_dict.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3783 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
