[issue46824] use AI_NUMERICHOST | AI_NUMERICSERV to skip getaddrinfo thread in asyncio
Ben Darnell added the comment: On MacOS in 2015, getaddrinfo was found to be much slower than inet_pton. Unless that's changed, this patch would be a performance regression on that platform. Data and benchmark script in https://groups.google.com/g/python-tulip/c/-SFI8kkQEj4/m/m1-oCMSABgAJ -- ___ Python tracker <https://bugs.python.org/issue46824> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46824] use AI_NUMERICHOST | AI_NUMERICSERV to skip getaddrinfo thread in asyncio
Ben Darnell added the comment: To summarize the justification, this patch does two things: it moves an optimization from create_connection to getaddrinfo, which makes it apply to more callers (including Tornado), and it makes the code simpler and less redundant (net reduction of 47 non-test lines in the patch). As far as we can tell, the reason it wasn't done this way in the first place is that at the time getaddrinfo held a global lock on some platforms, but this is no longer true. If there's still some locking in or around getaddrinfo on some platforms (or some libc implementations), this patch would be a bad idea. Is there a good way to test for that? I suppose we could set up a deliberately-slow DNS server and try to call getaddrinfo with AI_NUMERICHOST while another thread is blocked talking to that server, but that seems like a lot of test infrastructure to build out. -- nosy: +Ben.Darnell ___ Python tracker <https://bugs.python.org/issue46824> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Ben Darnell added the comment: > To be clear, by "cancel" you are not talking about Future.cancel(). Rather, > your handler causes all running tasks to finish (by sending a special message > on the socket corresponding to each running task). Is that right? Correct. My tasks here are calls to functions from the `select` module (one select call per executor task), and cancelling them means writing a byte to a pipe set up for this purpose. The select calls could be given a timeout so there is never an infinite task, but that's not ideal - a timeout that's too low has a performance cost as calls timeout and restart even when the system is "at rest", and a too-long timeout is still going to be perceived as a hanging application. > * it does not make sure the task associated with the socket finishes (no way > of knowing?) > * so if a task hangs while trying to stop then the running thread in the > ThreadPoolExecutor would block shutdown forever > * similarly, if a task is stuck handling a request then it will never receive > the special message on the socket, either blocking the send() in your handler > or causing ThreadPoolExecutor shutdown/atexit to wait forever Correct. If the task were buggy it could still cause a deadlock. In my case the task is simple enough (a single selector call) that this is not a risk. > * it vaguely implies a 1-to-1 relationship between sockets and *running* tasks > * likewise that pending (queued) tasks do not have an associated socket > (until started) Each task is associated with a selector object (managing a set of sockets), not a single socket. There is only ever one task at a time; a task is enqueued only after the previous one finishes. (This thread pool is not used for any other purpose) > * so once your handler finishes, any tasks pending in the ThreadPoolExecutor > queue will eventually get started but never get stopped by your handler; thus > you're back to the deadlock situation In my case this one-at-a-time rule means that the queue is always empty. But yes, in a more general solution you'd need some sort of interlock between cancelling existing tasks and starting new ones. > Alternately, perhaps ThreadPoolExecutor isn't the right fit here, as implied > by the route you ended up going. Yes, this is my conclusion as well. I filed this issue because I was frustrated that Python 3.9 broke previously-working code, but I'm willing to chalk this up to Hyrum's law and I'm not sure that this is something that ThreadPoolExecutor should be modified to support. -- ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39529] Deprecate get_event_loop()
Ben Darnell added the comment: > In IPython, I think you could use new_event_loop() for getting a new loop > instance. > Then, save the loop reference somewhere as a direct attribute, > threading.local or ContextVar. > Calling loop.run_until_complete() looks pretty normal in your situation. That works if you're a leaf in the dependency tree, but what about a library like Tornado (which IPython depends on)? If intermediate libraries (say Tornado and Twisted) introduce their own thread-local loops for backwards compatibility of their existing interfaces, the ecosystem gets fragmented again. Having the thread-local live in asyncio is useful for seamless interoperability across frameworks. > asyncio.run() is entirely self-reliant in that it creates all needed > resources at the start and closes them in finalization, rather than depending > on existing resources. I believe this to be significantly safer and better > guaranteed to function as intended, although perhaps at some cost to > convenience in cases like your own where there only needs to be one event > loop. Whether or not this is more likely to function as intended depends on assumptions about user expectations. In Tornado (and other async systems I have used in the past), it is the norm to set up handlers on an event loop while it is not running and then start it afterwards. The behavior of asyncio.run is surprising to me. Maybe I'm weird, but regardless of which behavior is surprising to the fewest people, my point is that this change is very disruptive to Tornado and most applications using it, contrary to the claim that the maintenance burden should be pretty low. I realize it may be too late to change course, but my preference would have been to resolve the conflict between get_event_loop and asyncio.run by making asyncio.run essentially an alias for asyncio.get_event_loop().run_until_complete. This would relax the isolation of asyncio.run, but that doesn't seem very important to me. The comparable idioms in Tornado (using the IOLoop.run_sync method) all work this way and I've never seen any confusion related to this or a situation in which creating a brand-new event loop in run_sync would be desirable. -- ___ Python tracker <https://bugs.python.org/issue39529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39529] Deprecate get_event_loop()
Ben Darnell added the comment: > The maintenance burden of the introduced deprecation should be pretty low. This is going to cause an unpleasant amount of churn in the Tornado community. It's been idiomatic (going back 12 years now) to do all your setup in synchronous code before starting the event loop. I'm tempted to restore the implicit loop creation in Tornado's `IOLoop.current()` (which is now essentially a wrapper around `asyncio.get_event_loop()`). > It leads to weird errors when get_event_loop() is called at import-time and > asyncio.run() is used for asyncio code execution. *Checks asyncio.run's implementation* Oh, I can see how that's a problem (and I've been giving people bad advice about the interchangeability of Tornado's IOLoop.run_sync and asyncio.run). But why does `asyncio.run` unconditionally create a new event loop instead of running on `asyncio.get_event_loop`? If asyncio.run used asyncio.get_event_loop it seems to me that we'd have the expected behavior and wouldn't have to make breaking changes to get_event_loop. > Low-level new_event_loop()/loop.run_until_complete() are still present to run > async code if top-level run() is not suitable for any reason. What if you're using run_forever instead of run_until_complete? (this is the common mode of operation for Tornado) There are solutions, of course (my preferred one is `await asyncio.Event().wait()`), but it's an extra conceptual hurdle to throw at users in a "hello world" example and this is why I've stuck with the model that uses (the equivalent of) run_forever instead of asyncio.run. -- nosy: +Ben.Darnell ___ Python tracker <https://bugs.python.org/issue39529> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37373] Configuration of windows event loop for libraries
Ben Darnell added the comment: > It's even slightly easier for tornado, which can reasonably set the > proactor-wrapper policy at IOLoop start time, which means > `asyncio.get_event_loop()` returns a loop with add_reader. But pyzmq doesn't > get invoked until an event loop is already running. That's not what I'm doing in Tornado; I don't change the policy or the result of get_event_loop. Instead, I call get_event_loop (only once) and wrap its result in AddThreadSelectorEventLoop. This works even while the event loop is already running (which is not an uncommon case; there is no expectation that you use tornado.ioloop.IOLoop.start instead of asyncio.EventLoop.run_forever). This relies on the fact that I already have my own thread-local lookup function to retrieve the wrapped event loop; an application that used the more typical asyncio patterns and relied on get_event_loop would indeed have difficulty with this pattern. -- ___ Python tracker <https://bugs.python.org/issue37373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32958] socket module calls with long host names can fail with idna codec error
Ben Darnell added the comment: [I'm coming here from https://github.com/tornadoweb/tornado/pull/3010) UnicodeError is a subclass of ValueError, so I don't see what value that change would provide. The thing that's surprising to me is that it's not a `socket.herror` (or `gaierror` for socket.getaddrinfo). I guess the docs don't formally say that `herror`/`gaierror` is the *only* possible error from these functions, but `gaierror` was the only error I was catching so the unexpected UnicodeError escaped the layer that was intended to handle it. I do think that in the special case of `getaddrinfo` with the `AI_NUMERICHOST` flag it should be handled differently: in that mode there is no network access necessary and it's reasonable to assume that the only possible error is a `gaierror` with `EAI_NONAME`. I'd like to at least see better documentation about what errors are possible from this family of functions. -- nosy: +Ben.Darnell ___ Python tracker <https://bugs.python.org/issue32958> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Ben Darnell added the comment: I have resolved my issue here by moving from ThreadPoolExecutor to a plain threading.Thread that I manage by hand (https://github.com/tornadoweb/tornado/commit/15832bc423c33c9280564770046dd6918f3a31b4). Therefore I no longer need this for myself and I leave it up to you to decide whether there's anything worth doing at this point. -- ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Ben Darnell added the comment: > IMO, a better practice would be providing those potentially infinite running > tasks a direct method of escape and invoking it before calling > executor.shutdown(), it would be a more reliable approach. Agreed, but the problem is that I'm in a library (so I don't control the main module), and the library's interface does not mandate any sort of explicit shutdown method. There is a shutdown method, but almost no one calls it, and it's never caused a problem until Python 3.9 changed things so it deadlocks. > My only concern is that it might be a potential foot-gun. If the user submits > an atexit hook that deadlocks, it might prevent threads from shutting down > safely prior to interpreter finalization. Yes, and that is exactly the problem. concurrent.futures submits an atexit hook whose behavior depends on application code, and through that I have inadvertently caused a deadlock. -- ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
New submission from Ben Darnell : I'm dealing with a subtle deadlock involving concurrent.futures.ThreadPoolExecutor, and my solution that worked in Python 3.8 broke with 3.9. I'm running some long-running (possibly infinite) tasks in the thread pool, and I cancel them in an `atexit` callback so that everything can shut down cleanly (before ThreadPoolExecutor joins all worker threads in its own `atexit` hook). Python 3.9 broke this due to https://bugs.python.org/issue39812. That change introduced a new atexit-like mechanism to the threading module and uses it where Python 3.8 used regular atexit. This means that ThreadPoolExecutor's atexit runs before mine, and since I never get a chance to cancel my tasks, it deadlocks. One way I can solve this is to move my own atexit function to `threading._register_atexit`, so my strawman proposal here is to make that function public and documented. On the other hand, even without the change in Python 3.9, my use of `atexit` smells like an abuse of implementation details in ThreadPoolExecutor (getting the atexit callbacks called in the right order was tricky when the concurrent.futures module started using lazy loading in Python 3.7). So I would welcome other suggestions about how to handle long-running but cancelable operations in a ThreadPoolExecutor at shutdown. One clean solution is to do the cancellation at the end of the main module instead of in an atexit hook. However, I'm doing this at a library so I don't have any way but atexit to ensure that this happens. Another option is to forego ThreadPoolExecutor entirely and manage the threads myself. My code in question is in a not-yet-released branch of Tornado: https://github.com/tornadoweb/tornado/blob/5913aa43ecfdaa76876fc57867062227b907b1dd/tornado/platform/asyncio.py#L57-L73 With the master branch of Tornado, Python 3.9, and Windows, `python -c "from tornado.httpclient import HTTPClient; c = HTTPClient()` reliably deadlocks at interpreter shutdown. -- messages: 378144 nosy: Ben.Darnell priority: normal severity: normal status: open title: Make threading._register_atexit public? versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39010] ProactorEventLoop raises unhandled ConnectionResetError
Ben Darnell added the comment: I've fixed the test and added some commentary about the different levels of clean shutdown we must do to quiet all the warnings: https://github.com/python/cpython/pull/22066 -- ___ Python tracker <https://bugs.python.org/issue39010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39010] ProactorEventLoop raises unhandled ConnectionResetError
Change by Ben Darnell : -- pull_requests: +21156 pull_request: https://github.com/python/cpython/pull/22066 ___ Python tracker <https://bugs.python.org/issue39010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39010] ProactorEventLoop raises unhandled ConnectionResetError
Ben Darnell added the comment: I can confirm that those warnings appear to be coming from the test I added here. I'm not sure how to interpret them, though - what does it mean for the main thread to be dangling? -- ___ Python tracker <https://bugs.python.org/issue39010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39010] ProactorEventLoop raises unhandled ConnectionResetError
Ben Darnell added the comment: I've posted a pull request with a test and fix: https://github.com/python/cpython/pull/22017. It's a more targeted fix than cmeyer's PR (which I didn't even notice until now due to unfamiliarity with the BPO UI) -- ___ Python tracker <https://bugs.python.org/issue39010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39010] ProactorEventLoop raises unhandled ConnectionResetError
Change by Ben Darnell : -- pull_requests: +21117 pull_request: https://github.com/python/cpython/pull/22017 ___ Python tracker <https://bugs.python.org/issue39010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39651] Exceptions raised by EventLoop.call_soon_threadsafe
Ben Darnell added the comment: No, this is unrelated to bpo-39010. -- ___ Python tracker <https://bugs.python.org/issue39651> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37373] Configuration of windows event loop for libraries
Ben Darnell added the comment: > Would it be acceptable for you to *require* use of uvloop when Tornado is > used with AsyncIO? How, exactly? Adding the dependency is no problem, but AFAIK I'd still be stuck with an import-time side effect to set the event loop policy (or a .pth file hack, I guess). Maybe an import-time side effect that chooses uvloop is better since it's a functional superset of the two default windows event loops, but setting the policy at import time still has its problems (What if another module has initialized the event loop before tornado is imported? What if someone also wants to set a custom policy to work around asyncio's strict "only on the main thread" defaults?) That's why I started this thread not asking for a proactor+selector hybrid event loop, but a better way to *configure* the event loop because policies aren't really doing the job in this case. > I considered using the `selectors` module directly, but it's not as simple as > it sounds. FWIW, I've reconsidered this. Treating SelectorEventLoop as a black box means you don't have enough control over synchronization and it's hard to avoid spurious wakeups and busy loops on the selector thread. I have a (broken) prototype using SelectorEventLoop that I plan to rewrite to call select.select directly. -- ___ Python tracker <https://bugs.python.org/issue37373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37373] Configuration of windows event loop for libraries
Ben Darnell added the comment: I considered using the `selectors` module directly, but it's not as simple as it sounds. Using the low-level interface means you need to also use a self-waker-pipe (or socket on windows) and manage a queue analogous to that used by `call_soon_threadsafe`. We already have two implementations of this pattern in asyncio with subtle differences between them (such as this one i just found: https://bugs.python.org/issue39651). In the end you'd have to duplicate a non-trivial portion of SelectorEventLoop. While there might be some efficiency gains to be had by working directly with the lower-level interface (and avoiding some redundancies between the two threads' event loops), I think the most robust/safest option is to use the well-tested SelectorEventLoop so that the only new code is what's needed to pass things back and forth between threads. -- ___ Python tracker <https://bugs.python.org/issue37373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37373] Configuration of windows event loop for libraries
Ben Darnell added the comment: I have an implementation of the selector-in-another-thread solution in https://github.com/tornadoweb/tornado/pull/2815. Is something like this worth considering for Python 3.9, or was Tornado the only project experiencing this pain and a tornado-specific solution is enough? -- ___ Python tracker <https://bugs.python.org/issue37373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39651] Exceptions raised by EventLoop.call_soon_threadsafe
New submission from Ben Darnell :
Proactor and selector event loops behave differently when call_soon_threadsafe
races with a concurrent call to loop.close(). In a selector event loop,
call_soon_threadsafe will either succeed or raise a RuntimeError("Event loop is
closed"). In a proactor event loop, it could raise this RuntimeError, but it
can also raise an AttributeError due to an unguarded access to self._csock.
https://github.com/python/cpython/blob/1ed61617a4a6632905ad6a0b440cd2cafb8b6414/Lib/asyncio/proactor_events.py#L785-L787
Comments in BaseSelectorEventLoop._write_to_self indicate that this is
deliberate, so the `csock is not None` check here should probably be copied to
the proactor event loop version.
https://github.com/python/cpython/blob/1ed61617a4a6632905ad6a0b440cd2cafb8b6414/Lib/asyncio/selector_events.py#L129-L136
I'd also accept an answer that the exact behavior of this race is undefined and
it's up to the application to either arrange for all calls to
call_soon_threadsafe to stop before closing the loop. However, I've had users
of Tornado argue that they use the equivalent of call_soon_threadsafe in
contexts where this coordination would be difficult, and I've decided that
tornado's version of this method would never raise, even if there is a
concurrent close. So if asyncio declines to specify which exceptions are
allowed in this case, tornado will need to add a blanket `except Exception:`
around calls to call_soon_threadsafe.
--
components: asyncio
messages: 362078
nosy: Ben.Darnell, asvetlov, yselivanov
priority: normal
severity: normal
status: open
title: Exceptions raised by EventLoop.call_soon_threadsafe
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue39651>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39010] ProactorEventLoop raises unhandled ConnectionResetError
Ben Darnell added the comment: I just spent some time digging into this. Each call to `run_forever` starts a call to `_loop_self_reading`, then attempts to cancel it before returning: https://github.com/python/cpython/blob/1ed61617a4a6632905ad6a0b440cd2cafb8b6414/Lib/asyncio/windows_events.py#L312-L325 The comment at line 321 is not entirely accurate: the future will not resolve in the future, but it may have *already* resolved, and added its callback to the call_soon queue. This callback will run if the event loop is restarted again. Since `_loop_self_reading` calls itself, this results in two copies of the "loop" running concurrently and stepping on each other's `_self_reading_futures`. This appears to be fairly harmless except for noise in the logs when only one of the loops is stopped cleanly. I believe the simplest fix is for `_loop_self_reading` to compare its argument to `self._self_reading_future` to determine if it is the "current" loop and if not, don't reschedule anything. -- nosy: +Ben.Darnell ___ Python tracker <https://bugs.python.org/issue39010> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37373] Configuration of windows event loop for libraries
Ben Darnell added the comment: Yeah, it's definitely a hack. The argument for it, at best, is "practicality beats purity". The solution is two simple lines, but those lines need to be repeated in every project that depends on Tornado and cares about Windows, now or in the future. How many projects have to be affected to justify a hack like this? I'm not sure, but without this hack the add_reader methods, and by extension Tornado, will remain a compatibility trap. -- ___ Python tracker <https://bugs.python.org/issue37373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37373] Configuration of windows event loop for libraries
Ben Darnell added the comment: > From my understanding, there is no issue for Tornado itself. If Jupiter > Notebook needs Tornado, Tornado needs selector event loop on Windows -- > Jupiter can install the proper loop. Right. I'm just advocating for something that would make the transition smoother than each project independently stumbling across this issue and adding their own patch (and then fielding support issues from users who have ended up with a combination of versions that doesn't quite work). This of course depends on how many affected projects there are; I know Jupyter is the biggest but they're not the only one. There's no more direct way, but a thread that does select() should work. In fact, ProactorEventLoop could simply spin up a SelectorEventLoop in another thread when one of the FD methods is called, and proxy the callbacks back and forth. def add_reader(self, fd, callback, *args): if not self.selector_started: self.start_selector() self.selector.call_soon(self.selector.add_reader, fd, lambda: self.call_soon(callback, *args)) def start_selector(self): self.selector = SelectorEventLoop() def target(): asyncio.set_event_loop(self.selector) self.selector.run_forever() thread = threading.Thread(target=target) thread.start() # Clean shutdown is left as an exercise for the reader. Unifying the two interfaces like this would be preferable to adding more complexity for configuration IMHO. -- ___ Python tracker <https://bugs.python.org/issue37373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37373] Configuration of windows event loop for libraries
New submission from Ben Darnell : On Windows there are two event loop implementions with different interfaces: The proactor event loop is missing the file descriptor family of methods (add_reader()), while the selector event loop has other limitations including missing support for pipes and subprocesses and generally lower scalability. (The default has changed from selector to proactor in Python 3.8). If an application requires the selector event loop, it can set the global event loop policy when it starts up. But what if a library requires the selector event loop? It wouldn't be appropriate for a library to set the event loop policy. The best I can do is document that "This library requires an event loop which supports the add_reader() method; to use this library on Windows (directly or indirectly) you must install the WindowsSelectorEventLoopPolicy." This places a burden on application developers that target Windows to examine all their transitive dependencies to see if any require selectorevent loops (and if any have conflicting requirements for proactor event loops, which are even less likely to be documented since this is now the default). Concretely, this is a concern for Tornado (which requires add_reader()) and applications in the scientific python community (including Jupyter) which depend on it. I know it's probably too late to do anything about this for 3.8, but it would be great if there were some better process to negotiate the right event loop. Some ideas (none of which are very satisfying): - A declarative marker in setup.cfg or similar indicating the required feature set for an event loop - A dummy package that could be depended on to indicate the same thing (and maybe this package could even use .pth hacks to change the default simply by being installed, but that seems like a hack too far) - Some sort of runtime registry: at import time, Tornado could call `asyncio.add_required_features(asyncio.ADD_READER)` and this would influence the default policy - Flags to `get_event_loop()` indicating the required features (passed through to the policy). -- components: asyncio messages: 346289 nosy: Ben.Darnell, asvetlov, yselivanov priority: normal severity: normal status: open title: Configuration of windows event loop for libraries type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue37373> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31122] SSLContext.wrap_socket() throws OSError with errno == 0
Ben Darnell added the comment: We have an easy reproduction of this "[Errno 0] Error" on the server side in https://github.com/tornadoweb/tornado/issues/2504#issuecomment-426782158 It is triggered by a connection from `nc -z` (which I think is doing a TCP handshake and shutting down the connection cleanly, but I'm not sure. It might just send an RST instead of the clean shutdown). On macos, I get SSL_ERROR_EOF (as expected), but on linux it raises an OSError with errno 0. (Note that the script as posted has a small mistake in that it is using a client-side SSLContext on the server side. The same error is seen when that mistake is fixed) I'm going to add "errno 0" to the list of errors that Tornado should swallow silently here, so if you're trying to reproduce this in the future use Tornado 5.1.1. -- nosy: +Ben.Darnell ___ Python tracker <https://bugs.python.org/issue31122> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34700] typing.get_type_hints doesn't know about typeshed
Ben Darnell added the comment: Yeah, I think that would work at least for the sphinx use case. It seems like a strange partially-degraded mode and anything that needs structured access to the annotation would still need typeshed, but just getting the string would probably be enough for a lot of applications. In fact, sphinx *only* wants the string (AFAICT), so a separate method like get_type_hints_as_string() which stringifies everything even if it can be resolved might be a better route than an option to get_type_hints(). -- ___ Python tracker <https://bugs.python.org/issue34700> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34700] typing.get_type_hints doesn't know about typeshed
New submission from Ben Darnell : Currently, most type annotations for the standard library are provided in typeshed rather than in the library itself. Not all consumers of type annotations are aware of typeshed, and this can cause problems. Specifically, Sphinx 1.8 accesses type hints via `typing.get_type_hints()`, which fails for some hints which are (implicitly) relying on typeshed. This currently manifests as failures to build Tornado's docs with Sphinx 1.8 (Tornado is using inline type annotations). Concretely, the issue involves the `concurrent.futures.Future` class. In typeshed, this class is a `Generic[T]`, but in the standard library it's just a normal class. Consider a function that returns a parameterized Future type: from concurrent.futures import Future def do_something_background() -> 'Future[int]': return executor.submit(do_something) Note that the type annotation is already a string. We can't use `Future[int]` directly because at runtime, the real Future type is not subscriptable. The string defers resolution of the subscript until something tries to access the annotation *as a type hint*. When mypy does this, it uses the typeshed definition of Future, so everything passes. But when Sphinx calls `typing.get_type_hints()` on this function, it fails: ``` >>> typing.get_type_hints(do_something_background) Traceback (most recent call last): File "", line 1, in File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/typing.py", line 1543, in get_type_hints value = _eval_type(value, globalns, localns) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/typing.py", line 350, in _eval_type return t._eval_type(globalns, localns) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/typing.py", line 245, in _eval_type eval(self.__forward_code__, globalns, localns), File "", line 1, in TypeError: 'type' object is not subscriptable ``` What can be done about this? I see a few approaches: 1. Require that any use of type hints consults typeshed. This would entail either making typing.get_type_hints aware of typeshed or deprecating and removing it. 2. Disallow references to typeshed from inline type annotations; types that require typeshed must only appear in `.pyi` stubs, which will only be interpreted by tools like mypy that know about typeshed. (is this true, or are there tools that know about pyi files but not typeshed?) 3. Annotate types in the standard library as generic. So far, this problem has always taken the form of "type is not subscriptable", and it could be resolved by making all generic types in the stdlib subclasses of `typing.Generic[T]`. This would bring enough typing detail into the stdlib to allow the annotations to be parsed without typeshed. This would also avoid the need for the awkward quotes in the example above. -- messages: 325455 nosy: Ben.Darnell priority: normal severity: normal status: open title: typing.get_type_hints doesn't know about typeshed type: enhancement versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue34700> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32038] Add API to intercept socket.close()
Ben Darnell <ben.darn...@gmail.com> added the comment: Note that a guard on socket objects can only solve part of the problem: in the case where i've seen this bug, it was with raw file descriptors from libcurl, and there's nothing python can do about that because there are no (python) socket objects. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32038> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32038] Add API to intercept socket.close()
Ben Darnell <ben.darn...@gmail.com> added the comment: It's worse than a resource leak - the same file descriptor number could be reused for a different file/socket, and then depending on the selector in use, you could see the data from a completely different connection. I did see a bug like this years ago (in libcurl), although it's not a common problem. I'd use the proposed hook if it existed, but it seems like an intrusive solution to a rare issue. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32038> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26141] typing module documentation incomplete
New submission from Ben Darnell: The typing module docs at https://docs.python.org/3/library/typing.html do not include everything that is documented in the PEP (https://www.python.org/dev/peps/pep-0484/#the-typing-module). Specifically, `AnyStr` is mentioned but not defined, the `@overload` decorator is missing, and so are the new-to-3.5 types Awaitable, AsyncIterable, AsyncIterator. -- assignee: docs@python components: Documentation messages: 258475 nosy: Ben.Darnell, docs@python priority: normal severity: normal status: open title: typing module documentation incomplete versions: Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26141> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Ben Darnell added the comment: I don't think operator.getfuture() is possible because there are multiple ways of turning an awaitable into a Future. asyncio has one way; tornado has another. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Ben Darnell added the comment: Yes, I can switch use the ABC instead, and I agree that it doesn't make sense to have the inspect method if it's going to be equivalent to the ABC. I'm happy with the outcome here but AFAIK the original issue still stands: the Awaitable ABC is unusual in that `isinstance(x, Awaitable)` may produce different results than `issubclass(x.__class__, Awaitable)`. I'd like to see more explicit documentation of the facts that A) functools.singledispatch is not always equivalent to isinstance() and B) Awaitable is one case (the only one in the stdlib?) where this distinction occurs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Ben Darnell added the comment: With the two changes I described things appear to be working, although I've only done light testing so far. For inspect.isgenerator(), my use case is here: https://github.com/tornadoweb/tornado/blob/2971e857104f8d02fa9107a0e13f50170eb4f30d/tornado/gen.py#L222 I currently do isinstance(x, types.GeneratorType), which will fail if x is actually a GeneratorWrapper. I can change this to use collections.abc.Generator when it exists, but then I'd have to have some conditional logic to switch between collections.abc and types depending on what version I'm running on. It would be nice if that were encapsulated in inspect.isgenerator(). More generally, the inconsistency between isgenerator() and iscoroutine() is kind of odd. I would expect that either all inspect functions or none of them would use a suitable ABC if one exists. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Ben Darnell added the comment: 4. While this patch addresses initial request from Ben only partially (generator-based coroutines still require __instancecheck__), A partial solution doesn't mean much to me: as long as the __instancecheck__ is sometimes necessary, I'll have to use inspect.iscoroutine all the time instead of using singledispatch with Awaitable. If anything, this just magnifies the risk of mysterious failures as things will work with async def but not yield from. I think I'd rather not have the ABC than have one with this kind of quirk. All this checking for coroutine-ness feels very strange to me. It's anti-duck-typing: all generators have all the methods necessary to satisfy the coroutine interface, but you can't use them as coroutines without some magical indication that that's what you meant. Prior to 3.5, a coroutine was just a callable returning a generator. Now, not only must it return a generator with the special coroutine flag, the callable itself must be of the right type, which causes problems when the underlying generator function is wrapped in complex ways (https://github.com/tornadoweb/tornado/blob/2971e857104f8d02fa9107a0e13f50170eb4f30d/tornado/testing.py#L476). Attempting to divide generators into awaitable and non-awaitable subsets is a complex solution to a problem that I'm not convinced is real. Was there a problem in practice with Python 3.4's asyncio in which people used yield from in a coroutine with generators that were intended to be iterators instead? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Ben Darnell added the comment: GeneratorWrapper helps, but it fails when applied to non-generator functions that return a value (while both tornado.gen.coroutine and asyncio.coroutine take pains to support such usage). The raise TypeError should be removed; just return the result without wrapping if it's not a generator. GeneratorWrapper also runs afoul of some places where I do explicit type checking instead of duck typing (isinstance(x, types.GeneratorType)). Using the Generator ABC doesn't work because the generator methods are set as attributes on __init__ instead of defined on the class. The methods should be defined on the class, even if they're overwritten with instance attributes later for speed. (related: inspect.iscoroutine is defined in terms of collections.abc.Coroutine. Should inspect.isgenerator be redefined to use the new collections.abc.Generator?) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
Ben Darnell added the comment: On Tue, Jun 9, 2015 at 10:12 PM, Yury Selivanov rep...@bugs.python.org wrote: Yury Selivanov added the comment: All this checking for coroutine-ness feels very strange to me. It's anti-duck-typing: [..] Why is it anti-duck-typing? Awaitable is an object that implements __await__. With this patch coroutines are a separate type with __await__ (although, ceval doesn't use it to make things faster). Anti-duck-typing isn't quite the right word. What I meant was that Python 3.4 had generators that were used as coroutines. When we introduced __await__ in 3.5, we could have added __await__ as a method for all generators, since all generators have the necessary send/throw/close methods. But we didn't, and now we require an explicit marker that a given object was intended for a particular use instead of inferring that intention from the set of methods available. In asyncio we check for coroutine-ness only to raise errors if someone passes a wrong object, or to make @asyncio.coroutine work (which will go away eventually). The check for coroutine-ness is not just in asyncio; it happens in the core interpreter. Tornado has to adapt to this check in order to interoperate with 'async def' coroutines. Now, not only must it return a generator with the special coroutine flag, the callable itself must be of the right type [..] Not sure what you mean here. It doesn't matter what callable is. It only matters if it returns a native coroutine, a generator-based coroutine, or an object with __await__. The type of the callable matters for the types.coroutine decorator. In order to get a coroutine object instead of a generator object, I must apply types.coroutine to the actual underlying generator, and not any wrapper. Attempting to divide generators into awaitable and non-awaitable subsets is a complex solution to a problem that I'm not convinced is real. Well, 'await' expression is a new operator, and it makes total sense to limit its usage only to awaitables. Awaiting on a generator that yields a fibonacci sequence just doesn't make any sense, and *is* error prone. I think it would be a major mistake to allow this just to make things a little bit more convenient during the transition period. This particular patch does not divide generators in awaitables and non-awaitables, it introduces a new type for 'async def' coroutines. All confusion with old generator-based coroutines and @coroutine decorator is here only because we try to be backwards compatible; the compatibility layer can be removed in 3.7 or 3.8. There are three eras of coroutines to consider: 'async def' in 3.5+, 'yield from' in 3.3-3.4, and 'yield' in 2.5+. I'm trying to provide a path from 'yield' to 'async def'; this compatibility layer will remain relevant as long as people are migrating from 2.7. Was there a problem in practice with Python 3.4's asyncio in which people used yield from in a coroutine with generators that were intended to be iterators instead? Yes, a lot of people were confused where they have coroutines and where they have generators, and this was even mentioned in the Rationale section of the PEP. I understand that yield-based coroutines can be confusing, but is this particular kind of confusion bad enough that if someone writes 'await fibonacci_generator()' we have to raise TypeError: object generator can't be used in 'await' expression instead of RuntimeError: Task got bad yield: 1? (that error message from asyncio could definitely be improved, but the fact that no one has improved it suggests that the problem isn't that severe) My goal here is to make it possible for Tornado applications running on 3.5 to use 'async def' and 'await' even though Tornado itself must remain compatible with 2.7 (as do many of the libraries that make up the Tornado ecosystem; I don't want libraries to have to update for compatibility with this feature). This is doable (I think) but non-trivial in the current design; making all generators awaitable would make it easier. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24400] Awaitable ABC incompatible with functools.singledispatch
New submission from Ben Darnell: The new collections.abc.Awaitable ABC relies on __instancecheck__, which makes it incompatible with functools.singledispatch (singledispatch works based on args[0].__class__; any instance-level information is discarded). This surprised me because the first thing I tried to do with Awaitable was add it to my singledispatch-based coroutine compatibility layer. Ideally coroutine would be an actual subclass of generator, instead of a generator with an extra bit set on the instance that changes how it answers isinstance() checks. That would be a big change, though, so it might be better to just document that Awaitable is kind of unusual (if we weren't already in the beta period I might argue that the ABCs should be removed and we should just use the functions in the inspect module instead). -- components: asyncio messages: 244942 nosy: Ben.Darnell, gvanrossum, haypo, yselivanov priority: normal severity: normal status: open title: Awaitable ABC incompatible with functools.singledispatch versions: Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24400 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23588] Errno conflicts in ssl.SSLError
Ben Darnell added the comment:
I agree that SSLError should have used a different attribute, but it's too late
for that - changing it would break any code currently relying on SSL errnos (in
particular asynchronous code using the SSL_ERROR_WANT_{READ,WRITE} error codes
for normal operation).
errno_scope is better than checking for isinstance(SSLError) because it lets
code that only wants posix errno to say so, instead of having to know about and
blacklist every OSError subclass that abuses the errno attribute (is SSLError
the only one?)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23588
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23588] Errno conflicts in ssl.SSLError
New submission from Ben Darnell: ssl.SSLError is a subclass of OSError but uses error codes that come from a different scope than the standard errno constants and may have conflicts. For example, on my system (OSX), ssl.SSL_ERROR_WANT_X509_LOOKUP and errno.EINTR have the same value. This would cause problems for code like the following: def f(): while True: try: return ssl_sock.read() except OSError as e: if e.errno == errno.EINTR: continue raise This function would loop endlessly on ssl.SSL_ERROR_WANT_X509_LOOKUP. (Note that I have not run into any problem like this in practice so this is not a severe issue; I just stumbled across the possibility). The EINTR case is moot now that Python itself retries on EINTR, but other conflicts are possible (the problem is also mitigated by the fact that most of the interesting errnos now have dedicated exception subtypes) To use OSError.errno correctly, it is currently necessary to check the exact type of the exception; it is not enough to do subclass matching as is usual for exceptions. To remedy this, I propose either changing the numeric constants in the SSL module (if these are not required to match constants defined in openssl), or adding an attribute to OSError like 'errno_scope' to be used when interpreting the errno field. -- messages: 237239 nosy: Ben.Darnell priority: normal severity: normal status: open title: Errno conflicts in ssl.SSLError type: behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23588 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21965] Add support for Memory BIO to _ssl
Ben Darnell added the comment: Looks good to me. I've added exarkun and glyph to the nosy list since Twisted's experience with PyOpenSSL may provide useful feedback even though Twisted will presumably stick with what they've got instead of switching to this new interface. -- nosy: +exarkun, glyph ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21965 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20951] SSLSocket.send() returns 0 for non-blocking socket
Ben Darnell added the comment: Giampaolo, where do you see that send() may return zero if the other side has closed? I've always gotten an error in that case (EPIPE) I vote -1 to adding a new flag to control whether it returns zero or raises and +0 to just fixing it in Python 3.5 (I don't think returning zero is an unreasonable thing to do; it's not obvious to me from send(2) that it is guaranteed to never return zero although I believe that to be the case). It'll break Tornado, but there will be plenty of time to get a fix out before then. If there were a convenient place to put a deprecation warning I'd vote to deprecate in 3.5 and fix in 3.6, but there's no good way for the application to signal that it expects a WANT_WRITE exception. Another option may be to have SSLSocket.send() convert the WANT_WRITE exception into a socket.error with errno EAGAIN. This wouldn't break Tornado and would make socket.send and SSLSocket.send more consistent, but it's weird to hide the true error like this. -- nosy: +Ben.Darnell ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20951 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17997] ssl.match_hostname(): sub string wildcard should not match IDNA prefix
Changes by Ben Darnell ben.darn...@gmail.com: -- nosy: +Ben.Darnell ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17997 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8240] ssl.SSLSocket.write may fail on non-blocking sockets
Ben Darnell added the comment: I vote for enabling SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER by default. It can catch mistakes (i.e. failure to check the return value of send) in Python just as easily as in C, but I don't think those mistakes are common enough to be worth the headache of this error. The false positive rate of this error is higher in Python than in C because we don't have direct control over memory and pointers. As for partial writes, I'm not sure if it's backwards compatible to turn them on by default, but it might be nice if the option were exposed. Partial writes may have less benefit in Python than in C since we'd have to reallocate and copy a string instead of just moving a pointer. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13721] ssl.wrap_socket on a connected but failed connection succeeds and .peer_certificate gives AttributeError
Ben Darnell added the comment: That proposal sounds good to me. I agree that any change here should target 3.4, not backports to older versions. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13721] ssl.wrap_socket on a connected but failed connection succeeds and .peer_certificate gives AttributeError
Ben Darnell added the comment: We found a related issue with Tornado: https://github.com/facebook/tornado/pull/750 We call wrap_socket with do_handshake_on_connect=False and then call do_handshake when the socket is ready. If the getpeername call in wrap_socket fails with ENOTCONN because the connection was reset immediately after it was opened, then do_handshake will fail with an AttributeError (because self._sslobj is None). do_handshake should fail with a clearer error (perhaps socket.error with ENOTCONN or ECONNABORTED?) if self._sslobj is None. Also, Mac OS X appears to return EINVAL from getpeername in this case, instead of ENOTCONN. wrap_socket should probably treat ENOTCONN and EINVAL from getpeername as equivalent. -- nosy: +Ben.Darnell ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13721 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8240] ssl.SSLSocket.write may fail on non-blocking sockets
Ben Darnell added the comment: Related pypy issue: https://bugs.pypy.org/issue1238 -- nosy: +Ben.Darnell ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15626] unittest.main negates -bb option and programmatic warning configuration
New submission from Ben Darnell: In python 3.2, unittest.main by default modifies the warning configuration if no -W options were given on the command line. This undoes the effect of -bb, turning BytesWarning back into a warning instead of an error. If both -bb and -Werror::BytesWarning are given, then unittest does not tamper with the warning configuration and byte operations raise errors as expected (but note that -Werror doesn't work in the current version of virtualenv due to https://github.com/pypa/virtualenv/issues/194 ). unittest.main should check for the -bb flag in addition to sys.warnoptions. Additionally, if warning filters were modified programmatically before calling unittest.main, unittest will still modify the configuration and potentially override the user's settings. This is a rarer case, but I've done it before to avoid the problem with -W in a virtualenv (https://github.com/facebook/tornado/blob/master/tornado/test/runtests.py). It would be good for there to be some way to tell unittest.main not to touch the warnings at all. main(warnings=False) currently works, but that's relying on an implementation detail (any non-None, non-truthy value works), so either this should be documented or an official way to do the same thing should be added. The attached test file demonstrates the problem: when run with -bb, an warning is logged, but the test passes. -- files: testbb.py messages: 168013 nosy: Ben.Darnell priority: normal severity: normal status: open title: unittest.main negates -bb option and programmatic warning configuration type: behavior versions: Python 3.2 Added file: http://bugs.python.org/file26771/testbb.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15626 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14208] No way to recover original argv with python -m
New submission from Ben Darnell ben.darn...@gmail.com: I have a script which attempts to re-invoke itself using sys.argv, but it fails when run with python -m package.module. The problem is that the handling of -m (via the runpy module) rewrites sys.argv as if it were run as python package/module.py, but the two command lines are not equivalent: With -m the current directory is inserted at the head of sys.path, but without -m it's the directory containing module.py. The net effect is that the initial run of python -m package.module works as expected, but when it re-runs itself as python package/module.py the imports from module.py are effectively relative instead of absolute. One possible solution would be to provide an immutable sys.__argv__ (by analogy with sys.__stdout__ and friends). -- messages: 155002 nosy: Ben.Darnell priority: normal severity: normal status: open title: No way to recover original argv with python -m type: enhancement ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue14208 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13747] ssl_version documentation error
New submission from Ben Darnell ben.darn...@gmail.com: The ssl module docs claim that the default ssl_version for client-side operation is SSLv3, but it is actually SSLv23. The exact behavior depends on the version of openssl: starting in 1.0 the connection is limited by default to SSLv3 or TLSv1 (as documented in the note below the compatibility table), but in older versions of openssl SSLv2 is allowed by default. This is just a documentation error if you've got a recent version of openssl, but it's also a security problem with older versions, since people may have been unknowingly using the weaker SSLv2 protocol. (I don't know how widespread pre-1.0 versions of openssl are these days, but OSX Lion still ships with 0.9.8) It would be nice if the default mode were SSLv23 with SSL_OP_NO_SSLv2 set so the defaults would be safe even with older versions of openssl (there's no way to set this configuration from python code before py3.2) Also, the compatibility table claims that an SSLv3 client can talk to an SSLv2 server, which is incorrect. SSLv23 clients can talk to SSLv3 and TLSv1 servers if openssl is at least version 1.0 and SSLv2 ciphers are not explicitly enabled. -- messages: 150963 nosy: Ben.Darnell priority: normal severity: normal status: open title: ssl_version documentation error versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13747 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13747] ssl_version documentation error
Ben Darnell ben.darn...@gmail.com added the comment: Not necessarily. If I want to run python 2.7 or 3.x on an older linux distribution (e.g. Ubuntu 10.04 LTS, which has python 2.6 and openssl 0.9.8), I need to build from source, but I wouldn't think to update/rebuild all the dependencies from the ground up. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13747 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11657] multiprocessing_{send,recv}fd fail with fds 256
Ben Darnell ben.darn...@gmail.com added the comment: These functions are used when passing a socket object across a multiprocessing.Queue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11657 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12529] cgi.parse_header fails on double quotes and semicolons
New submission from Ben Darnell ben.darn...@gmail.com:
cgi.parse_header doesn't work on headers that contain combinations of double
quotes and semicolons (although it works with either type of character
individually).
cgi.parse_header('form-data; name=files; filename=fo\\o;bar')
('form-data', {'name': 'files', 'filename': 'fo\\o'})
This issue is present in python 2.7 and 3.2. One solution is to change
_parseparam as follows (same as email.message._parseparam):
def _parseparam(s):
while s[:1] == ';':
s = s[1:]
end = s.find(';')
while end 0 and (s.count('', 0, end) - s.count('\\', 0, end)) % 2:
end = s.find(';', end + 1)
if end 0:
end = len(s)
f = s[:end]
yield f.strip()
s = s[end:]
--
messages: 140091
nosy: Ben.Darnell
priority: normal
severity: normal
status: open
title: cgi.parse_header fails on double quotes and semicolons
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12529
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11657] multiprocessing_{send,recv}fd fail with fds 256
New submission from Ben Darnell ben.darn...@gmail.com:
Line 125 of multiprocessing.c is *CMSG_DATA(cmsg) = fd;. CMSG_DATA returns
an unsigned char*, while fd is an int, so this code does not support file
descriptors 256 (additionally, I'm not sure if the buffer is guaranteed to be
initialized with zeros). recvfd has an analogous problem at line 168. Both
of these need to be changed to copy the entire integer, e.g. by casting the
result of CMSG_DATA to an int*.
http://hg.python.org/cpython/file/5deb2094f033/Modules/_multiprocessing/multiprocessing.c
--
messages: 131947
nosy: Ben.Darnell
priority: normal
severity: normal
status: open
title: multiprocessing_{send,recv}fd fail with fds 256
type: behavior
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11657
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
