[issue7946] Convoy effect with I/O bound threads and New GIL
Change by Charles-François Natali : -- nosy: -neologix ___ Python tracker <https://bugs.python.org/issue7946> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12545] os.lseek() and FileIO.seek() does not support offset larger than 2^63-1
Change by Charles-François Natali : -- nosy: -neologix ___ Python tracker <https://bugs.python.org/issue12545> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17263] crash when tp_dealloc allows other threads
Change by Charles-François Natali : -- nosy: -neologix ___ Python tracker <https://bugs.python.org/issue17263> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12488] multiprocessing.Connection does not communicate pipe closure between parent and child
Change by Charles-François Natali : -- nosy: -neologix ___ Python tracker <https://bugs.python.org/issue12488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can lose data from buffered files in reference cycles
Change by Charles-François Natali : -- nosy: -neologix ___ Python tracker <https://bugs.python.org/issue17852> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22367] Add open_file_descriptor parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)
Change by Charles-François Natali : -- nosy: -neologix ___ Python tracker <https://bugs.python.org/issue22367> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26692] cgroups support in multiprocessing
Charles-François Natali added the comment: I'm not convinced. The reason is that using the number of CPU cores is just a heuristic for a *default value*: the API allows the user to specify the number of workers to use, so it's not really a limitation. The problem is that if you try to think about a more "correct" default value, it gets complicated: here, it's about cgroups, but for example: - What if they are multiple processes running on the same box? - What if the process is subject to CPU affinity? Currently, the CPU affinity mask is ignored. - What if the code being executed by children is itself multi-threaded (maybe because it's using a numerical library using BLAS etc)? - What about hyper-threading? If the code has a lot of cache misses, it would probably be a good idea to use one worker per logical thread, but if it's cache-friendly, probably not. - Etc. In other words, I think that there's simply not reasonable default value for the number of workers to use, that any value will make some class of users/use-case unhappy, and it would add a lot of unnecessary complexity. Since the user can always specify the number of workers - if you find a place where it's not possible, then please report it - I really think we should let the choice/burden up to the user. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26692> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30014] Speedup DefaultSelectors.modify() by 2x
Charles-François Natali added the comment: The rationale for rejecting wouldn't be "DRY does not apply in this case", it would be that this makes the code more complicated, and that a negligible speedup would not be worth it. Now, thanks to your benchmark, a 10% speedup is not negligible, so that seems like a reasonable change. I'll have a look at your refactoring code, and once pushed, we can optimize modify(). -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30014> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30014] Speedup DefaultSelectors.modify() by 2x
Charles-François Natali added the comment: This refactoring was already suggested a long time ago, and at the time both Guido and I didn't like it because it makes the code actually more complicated: DRY in this case doesn't apply IMO. Also, this whole thread is a repeat of: http://bugs.python.org/issue18932 At the time, I already asked for one realistic use case demonstrating the benefit of implementing modify() natively instead of unregister()+register(). I know that this code makes modify() faster, but as I said multiple times, I'd like to se the impact on something else than a micro-benchmark: arguably if you propose this it's because you've profiled/found it to be an actual bottleneck, do you have *one* benchmark to support this change? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30014> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30014] Speedup DefaultSelectors.modify() by 2x
Charles-François Natali added the comment: > I don't think that selector.modify() can be a bottleneck, but IMHO the change > is simple and safe enough to be worth it. In a network server with 10k > client, an optimization making .modify() 1.52x faster is welcomed. IMHO it complicates the code for little benefit: that's why I asked for a realistic benchmark. This patch could made modify() 10x faster, if modify() only accounts for 1% of the overall overhead in a realistic use-case, then it's not worth it. -- title: Speedup DefaultSelectors.modify() by 1.5x -> Speedup DefaultSelectors.modify() by 2x ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30014> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30014] Speedup DefaultSelectors.modify() by 2x
Charles-François Natali added the comment: Hm, do you have a realistic benchmark which would show the benefit? Because this is really a micro-benchmark, and I'm not convinced that Selector.modify() is a significant bottleneck in a real-world application. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue30014> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29343] sock.close() raises OSError EBADF when socket's fd is closed
Charles-François Natali added the comment: FWIW I agree with Antoine and Martin: ignoring EBADF is a bad idea, quite dangerous. The man page probably says this to highlight that users shouldn't *retry* close(): """ Retrying the close() after a failure return is the wrong thing to do, since this may cause a reused file descriptor from another thread to be closed. This can occur because the Linux kernel always releases the file descriptor early in the close operation, freeing it for reuse; the steps that may return an error, such as flushing data to the filesystem or device, occur only later in the close operation. """ (Including on EINTR). In short, I think that the change is a good idea: Socket.close() is guaranteed to be idempotent, so this error can only happen in case of invalid usage. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue29343> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18966] Threads within multiprocessing Process terminate early
Charles-François Natali added the comment: One reason for not calling sys.exit() is because on Linux, the default implementation uses fork(), hence the address space in the chilren is a clone of the parent: so all atexit handlers, for example, would be called multiple times. There's also the problem that fork() isn't MT-safe, making the probability of screwups/deadlocks in various destructors/stack unwinding greater. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18966> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7105] weak dict iterators are fragile because of unpredictable GC runs
Charles-François Natali added the comment: Shouldn't the documentation be updated? https://docs.python.org/3.6/library/weakref.html#weakref.WeakKeyDictionary Note Caution: Because a WeakKeyDictionary is built on top of a Python dictionary, it must not change size when iterating over it. This can be difficult to ensure for a WeakKeyDictionary because actions performed by the program during iteration may cause items in the dictionary to vanish “by magic” (as a side effect of garbage collection). -- nosy: +neologix ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue7105> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26601] Use new madvise()'s MADV_FREE on the private heap
Charles-François Natali added the comment: The heap on Linux is still a linear contiguous *address space*. I agree that MADV_DONTNEED allow's returning committed memory back to the VM subsystem, but it is still using a large virtual memory area. Not everyone runs on 64-bit, or can waste address space. Also, not every Unix is Linux. But it might make sense to use malloc on Linux, maybe only on 64-bit. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26601] Use new madvise()'s MADV_FREE on the private heap
Charles-François Natali added the comment: > Julian Taylor added the comment: > > it defaulted to 128kb ten years ago, its a dynamic threshold since ages. Indeed, and that's what encouraged switching the allocator to use mmap. The problem with dynamic mmap threshold is that since the Python allocator uses fixed-size arenas, basically malloc always ends up allocating from the heap (brk). Which means that given that we don't use a - compacting - garbage collector, after a while the heap would end up quite fragmented, or never shrink: for example let's say you allocate 1GB - on the heap - and then you free them, but a single object is allocated at the top of the heap, you heap never shrinks back. This has bitten people (and myself a couple times at work). Now, I see several options: - revert to using malloc, but this will re-introduce the original problem - build some form of hysteresis in the arena allocation - somewhat orthogonally, I'd be interested to see if we couldn't increase the arena size -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23992] multiprocessing: MapResult shouldn't fail fast upon exception
Changes by Charles-François Natali <cf.nat...@gmail.com>: -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue23992> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Changes by Charles-François Natali <cf.nat...@gmail.com>: -- stage: needs patch -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue23530> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24303] OSError 17 due to _multiprocessing/semaphore.c assuming a one-to-one Pid -> process mapping.
Charles-François Natali added the comment: Anyone opposed to me committing the patch I submitted? It slves a real problem, and is fairly straight-forward (and conceptually more correct). -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24303> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24694] callables registered in TestCase.addCleanup should be run before tearDown
New submission from Charles-François Natali: Consider this code: - from __future__ import print_function from pyccp.unittest import SafeTestCase class MyTest(SafeTestCase): def setUp(self): print(setUp) def tearDown(self): print(tearDown) def test(self): print(creating) self.addCleanup(lambda: print(destroying)) - When run: setUp creating tearDown destroying We lose the LIFO ordering between between setUP and addCleanup, which is highly counter-intuitive, and almost always incorrect (despite addCleanup being docuemented to be run after tearDown). -- components: Library (Lib) messages: 247196 nosy: neologix priority: normal severity: normal status: open title: callables registered in TestCase.addCleanup should be run before tearDown type: behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24694 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24694] callables registered in TestCase.addCleanup should be run before tearDown
Charles-François Natali added the comment: I understand the risk of breakeage, but that's still broken, because we break LIFO ordering. I'd been using addCleanup() for years and never bothered looking at the documentation - which is admitedly a mistake - because LIFO ordering is the natural behavior: since addCleanup() is called once the test has been entered - so after setUp, logic would expect it to be undone before tearDown. But I guess that ship has sailed... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24694 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Charles-François Natali added the comment: It's just a doc improvement, I'm not convinced it really needs backporting. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Charles-François Natali added the comment: Committed. Julian, thanks for the patch! -- resolution: - fixed stage: - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23530 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23992] multiprocessing: MapResult shouldn't fail fast upon exception
Charles-François Natali added the comment: Barring any objections, I'll commit within the next few days. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23992 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24303] OSError 17 due to _multiprocessing/semaphore.c assuming a one-to-one Pid - process mapping.
Changes by Charles-François Natali cf.nat...@gmail.com: -- keywords: +needs review nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24303 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23992] multiprocessing: MapResult shouldn't fail fast upon exception
Changes by Charles-François Natali cf.nat...@gmail.com: -- keywords: +needs review nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23992 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24303] OSError 17 due to _multiprocessing/semaphore.c assuming a one-to-one Pid - process mapping.
Charles-François Natali added the comment: Here's a patch against 2.7 using _PyOS_URandom(): it should apply as-is to 3.3. -- keywords: +patch nosy: +neologix versions: +Python 3.3 Added file: http://bugs.python.org/file39679/mp_sem_race.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24303 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24427] subclass of multiprocessing Connection segfault upon attribute acces
New submission from Charles-François Natali:
The following segfaults in _PyObject_GenericGetAttrWithDict:
from socket import socketpair
from _multiprocessing import Connection
class Crash(Connection):
pass
_, w = socketpair()
Crash(w.fileno()).bar
#0 _PyObject_GenericGetAttrWithDict (dict=0xa6b001c, name=0xb7281020,
obj=0x8c12478) at Objects/object.c:1427
#1 PyObject_GenericGetAttr (obj=0x8c12478, name=0xb7281020) at
Objects/object.c:1461
(gdb) p *(obj + obj-ob_type-tp_dictoffset)
$8 = {ob_refcnt = 0, ob_type = 0x0}
It's probably not specific to this example, but I'm not familiar enough with
object construction/descriptors to know what's going on here.
Note that the following atch fixes the crash, but I don't know why:
--- a/Modules/_multiprocessing/connection.h Wed Apr 15 19:30:38 2015 +0100
+++ b/Modules/_multiprocessing/connection.h Wed Jun 10 20:25:15 2015 +0100
@@ -58,7 +58,7 @@
return NULL;
}
-self = PyObject_New(ConnectionObject, type);
+self = type-tp_alloc(type, 0);
if (self == NULL)
return NULL;
@@ -86,7 +86,7 @@
CLOSE(self-handle);
Py_END_ALLOW_THREADS
}
-PyObject_Del(self);
+Py_TYPE(self)-tp_free((PyObject*)self);
}
/*
--
messages: 245140
nosy: neologix, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: subclass of multiprocessing Connection segfault upon attribute acces
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24427
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23992] multiprocessing: MapResult shouldn't fail fast upon exception
Changes by Charles-François Natali cf.nat...@gmail.com: Removed file: http://bugs.python.org/file39171/mp_map_fail_fast_default.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23992 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23992] multiprocessing: MapResult shouldn't fail fast upon exception
Changes by Charles-François Natali cf.nat...@gmail.com: Removed file: http://bugs.python.org/file39170/mp_map_fail_fast_27.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23992 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23992] multiprocessing: MapResult shouldn't fail fast upon exception
Changes by Charles-François Natali cf.nat...@gmail.com: Added file: http://bugs.python.org/file39172/mp_map_fail_fast_27.diff Added file: http://bugs.python.org/file39173/mp_map_fail_fast_default.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23992 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23992] multiprocessing: MapResult shouldn't fail fast upon exception
Charles-François Natali added the comment: Patches for 2.7 and default. -- keywords: +patch Added file: http://bugs.python.org/file39170/mp_map_fail_fast_27.diff Added file: http://bugs.python.org/file39171/mp_map_fail_fast_default.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23992 ___diff -r 5576c8240963 Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py Wed Apr 15 19:30:38 2015 +0100 +++ b/Lib/multiprocessing/pool.py Wed Apr 22 19:34:48 2015 +0100 @@ -599,10 +599,10 @@ self._number_left = length//chunksize + bool(length % chunksize) def _set(self, i, success_result): +self._number_left -= 1 success, result = success_result -if success: +if success and self._success: self._value[i*self._chunksize:(i+1)*self._chunksize] = result -self._number_left -= 1 if self._number_left == 0: if self._callback: self._callback(self._value) @@ -615,15 +615,17 @@ self._cond.release() else: -self._success = False -self._value = result -del self._cache[self._job] -self._cond.acquire() -try: -self._ready = True -self._cond.notify() -finally: -self._cond.release() +if self._success: +self._success = False +self._value = result +if self._number_left == 0: +del self._cache[self._job] +self._cond.acquire() +try: +self._ready = True +self._cond.notify() +finally: +self._cond.release() # # Class whose instances are returned by `Pool.imap()` diff -r 5576c8240963 Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py Wed Apr 15 19:30:38 2015 +0100 +++ b/Lib/test/test_multiprocessing.py Wed Apr 22 19:34:48 2015 +0100 @@ -1123,6 +1123,12 @@ time.sleep(wait) return x*x + +def raise_large_valuerror(wait): +time.sleep(wait) +raise ValueError(x * 1024**2) + + class SayWhenError(ValueError): pass def exception_throwing_generator(total, when): @@ -1262,6 +1268,27 @@ p.close() p.join() +def test_map_no_failfast(self): +# Issue #23992: the fail-fast behaviour when an exception is raised +# during map() would make Pool.join() deadlock, because a worker +# process would fill the result queue (after the result handler thread +# terminated, hence not draining it anymore). + +t_start = time.time() + +with self.assertRaises(ValueError): +p = self.Pool(2) +try: +p.map(raise_large_valuerror, [0, 1]) +finally: +time.sleep(0.5) +p.close() +p.join() + +# check that we indeed waited for all jobs +self.assertGreater(time.time() - t_start, 0.9) + + def unpickleable_result(): return lambda: 42 diff -r 21ae8abc1af3 Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py Mon Apr 13 12:30:53 2015 -0500 +++ b/Lib/multiprocessing/pool.py Wed Apr 22 19:35:31 2015 +0100 @@ -638,22 +638,24 @@ self._number_left = length//chunksize + bool(length % chunksize) def _set(self, i, success_result): +self._number_left -= 1 success, result = success_result if success: self._value[i*self._chunksize:(i+1)*self._chunksize] = result -self._number_left -= 1 if self._number_left == 0: if self._callback: self._callback(self._value) del self._cache[self._job] self._event.set() else: -self._success = False -self._value = result -if self._error_callback: -self._error_callback(self._value) -del self._cache[self._job] -self._event.set() +if self._success: +self._success = False +self._value = result +if self._number_left == 0: +if self._error_callback: +self._error_callback(self._value) +del self._cache[self._job] +self._event.set() # # Class whose instances are returned by `Pool.imap()` diff -r 21ae8abc1af3 Lib/test/_test_multiprocessing.py --- a/Lib/test/_test_multiprocessing.py Mon Apr 13 12:30:53 2015 -0500 +++ b/Lib/test/_test_multiprocessing.py Wed Apr 22 19:35:31 2015 +0100 @@ -1660,6 +1660,10 @@ def mul(x, y): return x*y +def raise_large_valuerror(wait): +time.sleep(wait) +raise ValueError(x * 1024**2) + class SayWhenError(ValueError): pass def
[issue23992] multiprocessing: MapResult shouldn't fail fast upon exception
New submission from Charles-François Natali: hanger.py from time import sleep def hang(i): sleep(i) raise ValueError(x * 1024**2) The following code will deadlock on pool.close(): from multiprocessing import Pool from time import sleep from hanger import hang with Pool() as pool: try: pool.map(hang, [0,1]) finally: sleep(0.5) pool.close() pool.join() The problem is that when one of the tasks comprising a map result fails with an exception, the corresponding MapResult is removed from the result cache: def _set(self, i, success_result): success, result = success_result if success: [snip] else: self._success = False self._value = result if self._error_callback: self._error_callback(self._value) === del self._cache[self._job] self._event.set() === Which means that when the pool is closed, the result handler thread terminates right away, because it doesn't see any task left to wait for. Which means that it doesn't drain the result queue, and if some worker process is trying to write a large result to it (hence the large valuerrror to fill the socket/pipe buffer), it will hang, and the pool won't shut down (unless you call terminate()). Although I can see the advantage of fail-fast behavior, I don't think it's correct because it breaks the invariant where results won't be deleted from the cache until they're actually done. Also, the current fail-fast behavior breaks the semantics that the call only returns when it has completed. Returning while some jobs part of the map are still running is potentially very bad, e.g. if the user call retries the same call, assuming that all the jobs are done. Retrying jobs that are idempotent but not parallel execution-safe would break with the current code. The fix is trivial, use the same logic as in case of success to only signal failure when all jobs are done. I'll provide a patch if it seems sensible :-) -- components: Library (Lib) messages: 241404 nosy: neologix, pitrou priority: normal severity: normal status: open title: multiprocessing: MapResult shouldn't fail fast upon exception type: behavior versions: Python 2.7, Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23992 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23648] PEP 475 meta issue
Charles-François Natali added the comment: fstat_not_eintr.py: run this script from a NFS share and unplug the network cable, wait, replug. Spoiler: fstat() hangs until the network is back, CTRL+c or setitimer() don't interrupt it. You have to mount the share with the eintr option. Getting EINTR on a local FS is probably more challenging. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23648 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23753] Drop HAVE_FSTAT: require fstat() to compile/use Python
Charles-François Natali added the comment: +1 from me, fstat() has always been par of POSIX. It's really likely Python won't build anyway on such systems. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23753 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23753] Drop HAVE_FSTAT: require fstat() to compile/use Python
Charles-François Natali added the comment: Serhiy Storchaka added the comment: See also issue12082. Yes, but I don't think we want to clutter the code to support exotic niche platforms. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23753 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23618] PEP 475: handle EINTR in the socket module
Charles-François Natali added the comment: Could you regenerate your latest patch? It doesn't show properly in the review tool. Also, what's with the assert? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23618 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23648] PEP 475 meta issue
Charles-François Natali added the comment: Well, all the syscalls which can blocki can fail with EINTR, so all I:O related one. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23648 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23694] PEP 475: handle EINTR in fileutils.c
Charles-François Natali added the comment: LGTM. Note that dup() cannot fail with EINTR, it is non-blocking: dup2() can fail, because f the target FD is open, it has to close it, but not dup(). See e.g. this man page from my Debian: EINTR The dup2() or dup3() call was interrupted by a signal; see signal(7). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23694 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23646] PEP 475: handle EINTR in the time module, retry sleep()
Charles-François Natali added the comment: As for the change to select/poll/etc, IIRC Guido was opposed to it, that's why I didn't update them. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23646 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23618] PEP 475: handle EINTR in the socket module
Charles-François Natali added the comment: If EINTR is received during connect, the socket is unusable, that's why i didn't implement it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23618 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: @Victor: please commit. Would be nice to have a test for it; -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23530] os and multiprocessing.cpu_count do not respect cpuset/affinity
Charles-François Natali added the comment:
Well, we already expose CPU affinity:
import os
os.sched_getaffinity(0)
{0}
IMO the current implementation is sufficient (and talking about
overcommitting for CPU is a bit moot if you're using virtual machine
anyways).
The current documentation says:
Return the number of CPUs in the system. Returns None if undetermined.
Which to me is clear enough, although if you want to add an explicit
note that this doesn't take cpu affinity into account that wouldn't
hurt.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23530
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21998] asyncio: support fork
Charles-François Natali added the comment: @neologix: Would you be ok to add a *private* _at_fork() method to selectors classes in Python 3.4 to fix this issue? Not really: after fork(), you're hosed anyway: Q6 Will closing a file descriptor cause it to be removed from all epoll sets automatically? A6 Yes, but be aware of the following point. A file descriptor is a reference to an open file description (see open(2)). Whenever a descriptor is duplicated via dup(2), dup2(2), fcntl(2) F_DUPFD, or fork(2), a new file descriptor referring to the same open file description is created. An open file description continues to exist until all file descriptors referring to it have been closed. A file descriptor is removed from an epoll set only after all the file descriptors referring to the underlying open file description have been closed (or before if the descriptor is explicitly removed using epoll_ctl(2) EPOLL_CTL_DEL). This means that even after a file descriptor that is part of an epoll set has been closed, events may be reported for that file descriptor if other file descriptors referring to the same underlying file description remain open. What would you do with the selector after fork(): register the FDs in a new epoll, remove them? There's no sensible default behavior, and I'd rrather avoid polluting the code for this. If asyncio wants to support this, it can create a new selector and re-register everything it wants manually: there's a Selector.get_map() exposing all that's needed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21998 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18932] Optimize selectors.EpollSelector.modify()
Charles-François Natali added the comment: Does anyone have a realistic use case where modify() is actually a non-negligible part? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18932 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: Antoine Pitrou added the comment: Would it be possible to push the latest patch right now It's ok for me. Please watch the buildbots :) Cool, I'll push on Friday evening or Saturday. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18932] Optimize selectors.EpollSelector.modify()
Charles-François Natali added the comment: Well, I'd like to see at least one benchmark. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18932 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23351] socket.settimeout(5.0) does not have any effect
Charles-François Natali added the comment: It's a kernel bug closing (working fine on my Debian wheezy with a more recent kernel BTW). -- resolution: - third party status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23351] socket.settimeout(5.0) does not have any effect
Changes by Charles-François Natali cf.nat...@gmail.com: -- status: open - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23351 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: I just realized I didn't retry upon EINTR for open(): eintr-4.diff adds this, along with tests (using a fifo). Also, I added comments explaining why we don't retry upon close() (see e.g. http://lwn.net/Articles/576478/ and http://linux.derkeiler.com/Mailing-Lists/Kernel/2005-09/3000.html). -- Added file: http://bugs.python.org/file37935/eintr-4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___diff -r 7e6340e67618 Lib/_pyio.py --- a/Lib/_pyio.py Fri Jan 30 00:16:31 2015 +0100 +++ b/Lib/_pyio.py Fri Jan 30 22:08:27 2015 + @@ -1006,10 +1006,7 @@ current_size = 0 while True: # Read until EOF or until read() would block. -try: -chunk = self.raw.read() -except InterruptedError: -continue +chunk = self.raw.read() if chunk in empty_values: nodata_val = chunk break @@ -1028,10 +1025,7 @@ chunks = [buf[pos:]] wanted = max(self.buffer_size, n) while avail n: -try: -chunk = self.raw.read(wanted) -except InterruptedError: -continue +chunk = self.raw.read(wanted) if chunk in empty_values: nodata_val = chunk break @@ -1060,12 +1054,7 @@ have = len(self._read_buf) - self._read_pos if have want or have = 0: to_read = self.buffer_size - have -while True: -try: -current = self.raw.read(to_read) -except InterruptedError: -continue -break +current = self.raw.read(to_read) if current: self._read_buf = self._read_buf[self._read_pos:] + current self._read_pos = 0 @@ -1214,8 +1203,6 @@ while self._write_buf: try: n = self.raw.write(self._write_buf) -except InterruptedError: -continue except BlockingIOError: raise RuntimeError(self.raw should implement RawIOBase: it should not raise BlockingIOError) diff -r 7e6340e67618 Lib/distutils/spawn.py --- a/Lib/distutils/spawn.pyFri Jan 30 00:16:31 2015 +0100 +++ b/Lib/distutils/spawn.pyFri Jan 30 22:08:27 2015 + @@ -137,9 +137,6 @@ try: pid, status = os.waitpid(pid, 0) except OSError as exc: -import errno -if exc.errno == errno.EINTR: -continue if not DEBUG: cmd = executable raise DistutilsExecError( diff -r 7e6340e67618 Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py Fri Jan 30 00:16:31 2015 +0100 +++ b/Lib/multiprocessing/connection.py Fri Jan 30 22:08:27 2015 + @@ -365,10 +365,7 @@ def _send(self, buf, write=_write): remaining = len(buf) while True: -try: -n = write(self._handle, buf) -except InterruptedError: -continue +n = write(self._handle, buf) remaining -= n if remaining == 0: break @@ -379,10 +376,7 @@ handle = self._handle remaining = size while remaining 0: -try: -chunk = read(handle, remaining) -except InterruptedError: -continue +chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: @@ -595,13 +589,7 @@ self._unlink = None def accept(self): -while True: -try: -s, self._last_accepted = self._socket.accept() -except InterruptedError: -pass -else: -break +s, self._last_accepted = self._socket.accept() s.setblocking(True) return Connection(s.detach()) diff -r 7e6340e67618 Lib/multiprocessing/forkserver.py --- a/Lib/multiprocessing/forkserver.py Fri Jan 30 00:16:31 2015 +0100 +++ b/Lib/multiprocessing/forkserver.py Fri Jan 30 22:08:27 2015 + @@ -188,8 +188,6 @@ finally: os._exit(code) -except InterruptedError: -pass except OSError as e: if e.errno != errno.ECONNABORTED: raise @@ -230,13 +228,7 @@ data = b'' length = UNSIGNED_STRUCT.size while len(data) length: -while True: -try: -s = os.read(fd, length - len(data)) -except InterruptedError
[issue23351] socket.settimeout(5.0) does not have any effect
Charles-François Natali added the comment:
The way socket timeouts are implemented is by using select() to determine
whether the socket is ready for read/write. In this case, select() probably
marks the socket ready even though the queue is full, which later raises
EAGAIN.
Indeed, and this looks like a kernel bug.
Working as expected on a RHEL6:
$ python /tmp/test_unix_sock_timeout.py
('sending ', 0)
took 0.000s
('sending ', 1)
took 0.000s
('sending ', 2)
took 0.000s
('sending ', 3)
took 0.000s
('sending ', 4)
took 0.000s
('sending ', 5)
took 0.000s
('sending ', 6)
took 0.000s
('sending ', 7)
took 0.000s
('sending ', 8)
took 0.000s
('sending ', 9)
took 0.000s
('sending ', 10)
took 0.000s
('sending ', 11)
took 1.000s
Traceback (most recent call last):
File /tmp/test_unix_sock_timeout.py, line 17, in module
s.sendto(hello, SOCKNAME)
socket.timeout: timed out
About SO_SNDTIMEO and SO_RCVTIMEO, POSIX says it is implementation-defined
whether the SO_SNDTIMEO option can be set. Also, they would not necessarily
apply to other operations such as accept().
Exactly, the current way timeouts are implemented ar The Right Way, IMO.
--
Added file: http://bugs.python.org/file37919/test_unix_sock_timeout.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23351
___import itertools
import os
import socket
import time
SOCKNAME = /tmp/test.sock
if os.fork() == 0:
time.sleep(1)
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
s.settimeout(1)
for i in itertools.count():
print(sending , i)
t0 = time.time()
try:
s.sendto(hello, SOCKNAME)
finally:
print(took {:.3f}s.format(time.time() - t0))
else:
os.remove(SOCKNAME)
s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
s.bind(SOCKNAME)
os.waitpid(-1, 0)___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: With eintr-2.diff, fast!: Victory \°/. Instrumented test_send, 3 socket.send calls, many socket.recv_into calls: Yep, that's expected. I think we should keep the default socket buffer size: it increases the test coverage, and it's probably not worth trying to increase it to make the test a bit faster, especially since it spends so much time sleeping. Antoine, I'm now happy with the patch, so we'll be waiting for your decision on the PEP with Victor :-). -- Added file: http://bugs.python.org/file37910/eintr-3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___diff -r fe0fddd6fd21 Lib/_pyio.py --- a/Lib/_pyio.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/_pyio.py Thu Jan 29 22:20:01 2015 + @@ -1006,10 +1006,7 @@ current_size = 0 while True: # Read until EOF or until read() would block. -try: -chunk = self.raw.read() -except InterruptedError: -continue +chunk = self.raw.read() if chunk in empty_values: nodata_val = chunk break @@ -1028,10 +1025,7 @@ chunks = [buf[pos:]] wanted = max(self.buffer_size, n) while avail n: -try: -chunk = self.raw.read(wanted) -except InterruptedError: -continue +chunk = self.raw.read(wanted) if chunk in empty_values: nodata_val = chunk break @@ -1060,12 +1054,7 @@ have = len(self._read_buf) - self._read_pos if have want or have = 0: to_read = self.buffer_size - have -while True: -try: -current = self.raw.read(to_read) -except InterruptedError: -continue -break +current = self.raw.read(to_read) if current: self._read_buf = self._read_buf[self._read_pos:] + current self._read_pos = 0 @@ -1214,8 +1203,6 @@ while self._write_buf: try: n = self.raw.write(self._write_buf) -except InterruptedError: -continue except BlockingIOError: raise RuntimeError(self.raw should implement RawIOBase: it should not raise BlockingIOError) diff -r fe0fddd6fd21 Lib/distutils/spawn.py --- a/Lib/distutils/spawn.pySun Jan 18 11:17:39 2015 +0200 +++ b/Lib/distutils/spawn.pyThu Jan 29 22:20:01 2015 + @@ -137,9 +137,6 @@ try: pid, status = os.waitpid(pid, 0) except OSError as exc: -import errno -if exc.errno == errno.EINTR: -continue if not DEBUG: cmd = executable raise DistutilsExecError( diff -r fe0fddd6fd21 Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/multiprocessing/connection.py Thu Jan 29 22:20:01 2015 + @@ -365,10 +365,7 @@ def _send(self, buf, write=_write): remaining = len(buf) while True: -try: -n = write(self._handle, buf) -except InterruptedError: -continue +n = write(self._handle, buf) remaining -= n if remaining == 0: break @@ -379,10 +376,7 @@ handle = self._handle remaining = size while remaining 0: -try: -chunk = read(handle, remaining) -except InterruptedError: -continue +chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: @@ -595,13 +589,7 @@ self._unlink = None def accept(self): -while True: -try: -s, self._last_accepted = self._socket.accept() -except InterruptedError: -pass -else: -break +s, self._last_accepted = self._socket.accept() s.setblocking(True) return Connection(s.detach()) diff -r fe0fddd6fd21 Lib/multiprocessing/forkserver.py --- a/Lib/multiprocessing/forkserver.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/multiprocessing/forkserver.py Thu Jan 29 22:20:01 2015 + @@ -188,8 +188,6 @@ finally: os._exit(code) -except InterruptedError: -pass except OSError as e: if e.errno != errno.ECONNABORTED: raise @@ -230,13 +228,7 @@ data = b'' length
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: OK, it turns out the culprit was repeated calls to BytesIO.getvalue(), which forced large allocations upon reception of every message. The patch attached fixes this (without changing the socket buffer size). -- Added file: http://bugs.python.org/file37905/eintr-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___diff -r fe0fddd6fd21 Lib/_pyio.py --- a/Lib/_pyio.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/_pyio.py Thu Jan 29 07:59:47 2015 + @@ -1006,10 +1006,7 @@ current_size = 0 while True: # Read until EOF or until read() would block. -try: -chunk = self.raw.read() -except InterruptedError: -continue +chunk = self.raw.read() if chunk in empty_values: nodata_val = chunk break @@ -1028,10 +1025,7 @@ chunks = [buf[pos:]] wanted = max(self.buffer_size, n) while avail n: -try: -chunk = self.raw.read(wanted) -except InterruptedError: -continue +chunk = self.raw.read(wanted) if chunk in empty_values: nodata_val = chunk break @@ -1060,12 +1054,7 @@ have = len(self._read_buf) - self._read_pos if have want or have = 0: to_read = self.buffer_size - have -while True: -try: -current = self.raw.read(to_read) -except InterruptedError: -continue -break +current = self.raw.read(to_read) if current: self._read_buf = self._read_buf[self._read_pos:] + current self._read_pos = 0 @@ -1214,8 +1203,6 @@ while self._write_buf: try: n = self.raw.write(self._write_buf) -except InterruptedError: -continue except BlockingIOError: raise RuntimeError(self.raw should implement RawIOBase: it should not raise BlockingIOError) diff -r fe0fddd6fd21 Lib/distutils/spawn.py --- a/Lib/distutils/spawn.pySun Jan 18 11:17:39 2015 +0200 +++ b/Lib/distutils/spawn.pyThu Jan 29 07:59:47 2015 + @@ -137,9 +137,6 @@ try: pid, status = os.waitpid(pid, 0) except OSError as exc: -import errno -if exc.errno == errno.EINTR: -continue if not DEBUG: cmd = executable raise DistutilsExecError( diff -r fe0fddd6fd21 Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/multiprocessing/connection.py Thu Jan 29 07:59:47 2015 + @@ -365,10 +365,7 @@ def _send(self, buf, write=_write): remaining = len(buf) while True: -try: -n = write(self._handle, buf) -except InterruptedError: -continue +n = write(self._handle, buf) remaining -= n if remaining == 0: break @@ -379,10 +376,7 @@ handle = self._handle remaining = size while remaining 0: -try: -chunk = read(handle, remaining) -except InterruptedError: -continue +chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: @@ -595,13 +589,7 @@ self._unlink = None def accept(self): -while True: -try: -s, self._last_accepted = self._socket.accept() -except InterruptedError: -pass -else: -break +s, self._last_accepted = self._socket.accept() s.setblocking(True) return Connection(s.detach()) diff -r fe0fddd6fd21 Lib/multiprocessing/forkserver.py --- a/Lib/multiprocessing/forkserver.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/multiprocessing/forkserver.py Thu Jan 29 07:59:47 2015 + @@ -188,8 +188,6 @@ finally: os._exit(code) -except InterruptedError: -pass except OSError as e: if e.errno != errno.ECONNABORTED: raise @@ -230,13 +228,7 @@ data = b'' length = UNSIGNED_STRUCT.size while len(data) length: -while True: -try: -s = os.read(fd, length - len(data)) -except InterruptedError: -pass -else: -break +s = os.read
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: eintr-1.diff doesn't seem to make any significant difference from eintr.diff on my system. It's still pegging a CPU at 100% and takes 7 minutes wall time to complete. Alright, enough played: the patch attached uses a memoryview and socket.recv_into() to remove all memory allocations: if this doesn't fix your problem, I don't know what's going on... -- Added file: http://bugs.python.org/file37909/eintr-2.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___diff -r fe0fddd6fd21 Lib/_pyio.py --- a/Lib/_pyio.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/_pyio.py Thu Jan 29 19:53:06 2015 + @@ -1006,10 +1006,7 @@ current_size = 0 while True: # Read until EOF or until read() would block. -try: -chunk = self.raw.read() -except InterruptedError: -continue +chunk = self.raw.read() if chunk in empty_values: nodata_val = chunk break @@ -1028,10 +1025,7 @@ chunks = [buf[pos:]] wanted = max(self.buffer_size, n) while avail n: -try: -chunk = self.raw.read(wanted) -except InterruptedError: -continue +chunk = self.raw.read(wanted) if chunk in empty_values: nodata_val = chunk break @@ -1060,12 +1054,7 @@ have = len(self._read_buf) - self._read_pos if have want or have = 0: to_read = self.buffer_size - have -while True: -try: -current = self.raw.read(to_read) -except InterruptedError: -continue -break +current = self.raw.read(to_read) if current: self._read_buf = self._read_buf[self._read_pos:] + current self._read_pos = 0 @@ -1214,8 +1203,6 @@ while self._write_buf: try: n = self.raw.write(self._write_buf) -except InterruptedError: -continue except BlockingIOError: raise RuntimeError(self.raw should implement RawIOBase: it should not raise BlockingIOError) diff -r fe0fddd6fd21 Lib/distutils/spawn.py --- a/Lib/distutils/spawn.pySun Jan 18 11:17:39 2015 +0200 +++ b/Lib/distutils/spawn.pyThu Jan 29 19:53:06 2015 + @@ -137,9 +137,6 @@ try: pid, status = os.waitpid(pid, 0) except OSError as exc: -import errno -if exc.errno == errno.EINTR: -continue if not DEBUG: cmd = executable raise DistutilsExecError( diff -r fe0fddd6fd21 Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/multiprocessing/connection.py Thu Jan 29 19:53:06 2015 + @@ -365,10 +365,7 @@ def _send(self, buf, write=_write): remaining = len(buf) while True: -try: -n = write(self._handle, buf) -except InterruptedError: -continue +n = write(self._handle, buf) remaining -= n if remaining == 0: break @@ -379,10 +376,7 @@ handle = self._handle remaining = size while remaining 0: -try: -chunk = read(handle, remaining) -except InterruptedError: -continue +chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: @@ -595,13 +589,7 @@ self._unlink = None def accept(self): -while True: -try: -s, self._last_accepted = self._socket.accept() -except InterruptedError: -pass -else: -break +s, self._last_accepted = self._socket.accept() s.setblocking(True) return Connection(s.detach()) diff -r fe0fddd6fd21 Lib/multiprocessing/forkserver.py --- a/Lib/multiprocessing/forkserver.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/multiprocessing/forkserver.py Thu Jan 29 19:53:06 2015 + @@ -188,8 +188,6 @@ finally: os._exit(code) -except InterruptedError: -pass except OSError as e: if e.errno != errno.ECONNABORTED: raise @@ -230,13 +228,7 @@ data = b'' length = UNSIGNED_STRUCT.size while len(data) length: -while True: -try: -s
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: I added a few prints to the send and receive loops of _test_send. When running on a reasonably current Debian testing Linux: Thanks, that's what I was suspecting, but I really don't understand why 200ms isn't enough for a socket write to actually do something: maybe OS-X and *BSD schedulers are large timeslice... Could you try by increasing signal_period to e.g. 0.5, and sleep_time to 1? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: Charles-François Natali added the comment: Hmmm... Basically, with a much smaller socket buffer, we get much more context switches, which increases drastically the test runtime. But I must admit I'm still really surprised by the time it takes on OS-X: with a SOCK_MAX_SIZE of 16MB and a socket buffer size of 8kb, that's 2000 calls to send with context switches between the sender and receiver - and thie overhead should be much less on a two core machine. OK, actually the receiver is completely CPU-bound, because of memory allocation for the socket.recv() buffer I'll play with recv_into() and profile a bit more. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: It turns out the times are not important; the hangup is the default size of the socket buffers on OS X and possibly BSD in general. In my case, the send and receive buffers are 8192, which explains why the chunks written are so small. Hmmm... Basically, with a much smaller socket buffer, we get much more context switches, which increases drastically the test runtime. But I must admit I'm still really surprised by the time it takes on OS-X: with a SOCK_MAX_SIZE of 16MB and a socket buffer size of 8kb, that's 2000 calls to send with context switches between the sender and receiver - and thie overhead should be much less on a two core machine. I'll try to increase the socket buffer size and see what the buildbots do: most of them will probably cap it at around 100k, but that'll hopefully be enough. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23302] Small fixes around the use of TCP MSS in http.client
Charles-François Natali added the comment: Or we should acknowledge that this is overkill, and take the same approach as all major web browser: disable the Nagle algorithm. For a protocol like http which is transaction oriented it's probably the best thing to do. -- nosy: +neologix, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23302 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23267] multiprocessing pool.py doesn't close inqueue and outqueue pipes on termination
Charles-François Natali added the comment: Interestingly, there is no close() method on SimpleQueue... -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23267 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR handling
Charles-François Natali added the comment: The review diff is weird: it seems it contains changes that aren't EINTR-related (see e.g. argparse.rst). Here's a manually generated diff. -- Added file: http://bugs.python.org/file37802/eintr.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___diff -r fe0fddd6fd21 Lib/_pyio.py --- a/Lib/_pyio.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/_pyio.py Wed Jan 21 05:55:58 2015 + @@ -1006,10 +1006,7 @@ current_size = 0 while True: # Read until EOF or until read() would block. -try: -chunk = self.raw.read() -except InterruptedError: -continue +chunk = self.raw.read() if chunk in empty_values: nodata_val = chunk break @@ -1028,10 +1025,7 @@ chunks = [buf[pos:]] wanted = max(self.buffer_size, n) while avail n: -try: -chunk = self.raw.read(wanted) -except InterruptedError: -continue +chunk = self.raw.read(wanted) if chunk in empty_values: nodata_val = chunk break @@ -1060,12 +1054,7 @@ have = len(self._read_buf) - self._read_pos if have want or have = 0: to_read = self.buffer_size - have -while True: -try: -current = self.raw.read(to_read) -except InterruptedError: -continue -break +current = self.raw.read(to_read) if current: self._read_buf = self._read_buf[self._read_pos:] + current self._read_pos = 0 @@ -1214,8 +1203,6 @@ while self._write_buf: try: n = self.raw.write(self._write_buf) -except InterruptedError: -continue except BlockingIOError: raise RuntimeError(self.raw should implement RawIOBase: it should not raise BlockingIOError) diff -r fe0fddd6fd21 Lib/distutils/spawn.py --- a/Lib/distutils/spawn.pySun Jan 18 11:17:39 2015 +0200 +++ b/Lib/distutils/spawn.pyWed Jan 21 05:55:58 2015 + @@ -137,9 +137,6 @@ try: pid, status = os.waitpid(pid, 0) except OSError as exc: -import errno -if exc.errno == errno.EINTR: -continue if not DEBUG: cmd = executable raise DistutilsExecError( diff -r fe0fddd6fd21 Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/multiprocessing/connection.py Wed Jan 21 05:55:58 2015 + @@ -365,10 +365,7 @@ def _send(self, buf, write=_write): remaining = len(buf) while True: -try: -n = write(self._handle, buf) -except InterruptedError: -continue +n = write(self._handle, buf) remaining -= n if remaining == 0: break @@ -379,10 +376,7 @@ handle = self._handle remaining = size while remaining 0: -try: -chunk = read(handle, remaining) -except InterruptedError: -continue +chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: @@ -595,13 +589,7 @@ self._unlink = None def accept(self): -while True: -try: -s, self._last_accepted = self._socket.accept() -except InterruptedError: -pass -else: -break +s, self._last_accepted = self._socket.accept() s.setblocking(True) return Connection(s.detach()) diff -r fe0fddd6fd21 Lib/multiprocessing/forkserver.py --- a/Lib/multiprocessing/forkserver.py Sun Jan 18 11:17:39 2015 +0200 +++ b/Lib/multiprocessing/forkserver.py Wed Jan 21 05:55:58 2015 + @@ -188,8 +188,6 @@ finally: os._exit(code) -except InterruptedError: -pass except OSError as e: if e.errno != errno.ECONNABORTED: raise @@ -230,13 +228,7 @@ data = b'' length = UNSIGNED_STRUCT.size while len(data) length: -while True: -try: -s = os.read(fd, length - len(data)) -except InterruptedError: -pass -else: -break +s = os.read(fd, length - len(data)) if not s: raise
[issue23285] PEP 475 - EINTR hanndling
Changes by Charles-François Natali cf.nat...@gmail.com: -- components: Library (Lib) hgrepos: 293 nosy: haypo, neologix, pitrou priority: normal severity: normal status: open title: PEP 475 - EINTR hanndling type: enhancement ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR hanndling
Changes by Charles-François Natali cf.nat...@gmail.com: -- keywords: +patch Added file: http://bugs.python.org/file37797/ff1274594739.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR hanndling
New submission from Charles-François Natali: The test runs fine on Linux, but hangs in test_send() on OS-X and *BSD. I don't know what's wrong, so if someone with access to one of these OS could have a look, it would be great. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23285] PEP 475 - EINTR handling
Changes by Charles-François Natali cf.nat...@gmail.com: -- title: PEP 475 - EINTR hanndling - PEP 475 - EINTR handling ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23285 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23225] selectors: raise an exception if the selector is closed
Charles-François Natali added the comment: RuntimeError sounds better to me (raising ValueError when no value is provided, e.g. in select() sounds definitely strange). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23225 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23009] selectors.EpollSelector.select raises exception when nothing to select.
Charles-François Natali added the comment: Thanks for taking care of this. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23009 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17852] Built-in module _io can loose data from buffered files at exit
Charles-François Natali added the comment: Serhiy, I believe this still happens in Python 3.4, but it is harder to reproduce. I couldn't get Armin's script to produce the problem either, but I'm pretty sure that this is what causes e.g. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=771452#60. Maybe, as it's been said several times, it's an application bug: files should be closed explicitly (or better with a context manager of course). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17852 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22697] Deadlock with writing to stderr from forked process
Charles-François Natali added the comment: Maries Ionel Cristian added the comment: Serhiy, I don't think this is a duplicate. Odd that you closed this without any explanation. This happens in a internal lock in cpython's runtime, while the other bug is about locks used in the logging module (which are very different). Yes, this is a duplicate. Whether the lock is an internal or a user-created one doesn't change anything, it's always the same problem of having a lock locked by another thread at the time of the fork(). -- resolution: - duplicate status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22697 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22698] Add constants for ioctl request codes
Charles-François Natali added the comment: Adding ioctl constants is fine. However, I think that if we do this, it'd be great if we could also expose this information in a module (since psutil inclusion was discussed recently), but that's probably another issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22698 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22631] Feature Request CAN_RAW_FD_FRAME
Charles-François Natali added the comment: Annoying. I thought CAN_RAW_FD_FRAME would be a macro, which would have made conditional compilation easy, but it's apparently a enum value, which means we have to add a configure-time check... -- components: +Library (Lib) -IO ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22631 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18643] add a fallback socketpair() implementation to the socket module
Changes by Charles-François Natali cf.nat...@gmail.com: -- resolution: - fixed stage: patch review - resolved status: open - closed title: add a fallback socketpair() implementation in test.support - add a fallback socketpair() implementation to the socket module ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18643 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17293] uuid.getnode() MAC address on AIX
Charles-François Natali added the comment: My only comment would be to use subprocess instead of os.popen(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17293 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17293] uuid.getnode() MAC address on AIX
Charles-François Natali added the comment: Why is that a different issue? The code you *add in this patch* uses os.popen, why not use subprocess instead? Furthermore, the code catches OSError when calling popen(), but popen() doesn't raise an exception. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17293 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22181] os.urandom() should use Linux 3.17 getrandom() syscall
Charles-François Natali added the comment: Note that I'm not fussed about it: far from simplifying the code, it will make it more complex, thus more error-prone. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22181 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21963] 2.7.8 backport of Issue1856 (avoid daemon thread problems at shutdown) breaks ceph
Charles-François Natali added the comment: Agreed with Antoine and Benjamin. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21963 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17835] test_io broken on PPC64 Linux
Charles-François Natali added the comment: Let's try with this instead: from socket import socket, SO_SNDBUF, SOL_SOCKET s = socket() s.getsockopt(SOL_SOCKET, SO_SNDBUF) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17835 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17835] test_io broken on PPC64 Linux
Charles-François Natali added the comment: In this case, the issues are being caused by the following kernel parameters that we have for our default build - # ## TIBCO network tuning # # net.core.rmem_default = 33554432 net.core.wmem_default = 33554432 net.core.rmem_max = 33554432 net.core.wmem_max = 33554432 Toggling the support.PIPE_MAX_SIZE to +32MB or temporarily removing these parameters mitigates the issue. Is there a better way of calculating support.PIPE_MAX_SIZE so it is reflective of the actual OS value? What does: import fcntl, os r, w = os.pipe() fcntl.fcntl(w, 1032) return? Note that the kernel buffer sizes above are, well, *really huge*. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17835 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22435] socketserver.TCPSocket leaks socket to garbage collector if server_bind() fails
Charles-François Natali added the comment: Patch attached. The test wouldn't result in FD exhaustion on CPython because of the reference counting, but should still trigger RessourceWarning. -- keywords: +patch nosy: +haypo, pitrou stage: - patch review Added file: http://bugs.python.org/file36665/socketserver_bind_leak.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22435 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22378] SO_MARK support for Linux
Charles-François Natali added the comment: Thanks, I committed a simpler version of the patch. -- resolution: - fixed stage: test needed - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22378 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22127] performance regression in socket getsockaddrarg()
Charles-François Natali added the comment: Note that even the bytes version is still quite slow. UDP is used for light-weight protocols where you may send thousands or more messages per second. I'd be curious what the sendto() performance is in raw C. Ah, I wouldn't rely on the absolyte values, my computer is *slow*. On a more recent machine, I get this: 10 loops, best of 3: 8.82 usec per loop Whereas a C loop gives a 4usec per loop. Abc is a bytes string in Python 2 and an Unicode string in Python 3. Sure, but why do getaddrinfo() and gethostbyname() return strings then? This means that someone using: addr = getaddrinfo(...) sendto(DATA, addr) Will pay the idna encoding upon every call to sendto(). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22127 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22127] performance regression in socket getsockaddrarg()
Charles-François Natali added the comment: Parsing a bytes object i.e. b'127.0.0.1' is done by inet_pton(), so it's probably cheap (compared to a syscall). If we had getaddrinfo() and gethostbyname() return bytes instead of strings, it would be a huge gain. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22127 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22127] performance regression in socket getsockaddrarg()
Charles-François Natali added the comment: Charles-François: you get the idna overhead in 2.7, too, by specifying u'127.0.0.1' as the address. I don't see it in a profile output, and the timing doesn't change whether I pass '127.0.0.1' or b'127.0.0.1' in 2.7. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22127 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22127] performance regression in socket getsockaddrarg()
Charles-François Natali added the comment: Please understand that Victor and I were asking you to pass a *unicode* object, with a *u* prefix. For me, the time more-than-doubles, on OSX, with the system python. Sorry, I misread 'b'. it's a day without... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22127 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22127] performance regression in socket.getsockaddr()
New submission from Charles-François Natali:
I noticed that socket.sendto() got noticably slower since 3.4 (at least),
compared to 2.7:
2.7:
$ ./python -m timeit -s import socket; s = socket.socket(socket.AF_INET,
socket.SOCK_DGRAM); DATA = b'hello'; TARGET=('127.0.0.1', 4242)
s.sendto(DATA, TARGET)
10 loops, best of 3: 15.8 usec per loop
3.4:
$ ./python -m timeit -s import socket; s = socket.socket(socket.AF_INET,
socket.SOCK_DGRAM); DATA = b'hello'; TARGET=('127.0.0.1', 4242)
s.sendto(DATA, TARGET)
1 loops, best of 3: 25.9 usec per loop
A profile reveals this:
2.7:
100065 function calls in 2.268 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
10.3610.3612.2682.268 test_send.py:1(module)
101.8950.0001.8950.000 {method 'sendto' of
'_socket.socket' objects}
3.4:
906015 function calls (905975 primitive calls) in 6.132 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
5/10.0000.0006.1326.132 {built-in method exec}
10.3340.3346.1326.132 test_send.py:1(module)
102.3470.0005.7690.000 {method 'sendto' of
'_socket.socket' objects}
101.9910.0003.4110.000 idna.py:147(encode)
5000860.8940.0000.8940.000 {built-in method len}
100.2690.0000.2690.000 {method 'encode' of 'str' objects}
100.2570.0000.2570.000 {method 'split' of 'bytes'
objects}
As can be seen, it's the IDNA encoding which takes a long time, and doesn't
appear in the 2.7 profile.
The parsing code (including idna codec) is present in both versions though:
static int
getsockaddrarg(PySocketSockObject *s, PyObject *args,
struct sockaddr *addr_ret, int *len_ret)
[...]
if (!PyArg_ParseTuple(args, eti:getsockaddrarg,
idna, host, port))
return 0;
I'm currently bisecting the commit, but I'm not familiar with encoding stuff.
Is it expected that the IDNA encoding be called when passed an ascii string?
--
components: Library (Lib)
messages: 224618
nosy: haypo, loewis, neologix, pitrou
priority: normal
severity: normal
status: open
title: performance regression in socket.getsockaddr()
type: performance
versions: Python 3.4, Python 3.5
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22127
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22127] performance regression in socket getsockaddrarg()
Changes by Charles-François Natali cf.nat...@gmail.com: -- title: performance regression in socket.getsockaddr() - performance regression in socket getsockaddrarg() ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22127 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22127] performance regression in socket getsockaddrarg()
Charles-François Natali added the comment:
For Python, the encoder is only used when you pass a Unicode string.
Hm...
I'm passing ('127.0.0.1', 4242)as destination, and you can see in the
above profile that the idna encode function is called.
This doesn't occur with 2.7.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22127
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22127] performance regression in socket getsockaddrarg()
Charles-François Natali added the comment:
OK, I think I see what you mean:
$ ./python -m timeit -s import socket; s =
socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.sendto(b'hello',
('127.0.0.1', 4242))1 loops, best of 3: 44.7 usec per loop
$ ./python -m timeit -s import socket; s =
socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.sendto(b'hello',
(b'127.0.0.1', 4242))
1 loops, best of 3: 23.7 usec per loop
That's really surprising, especially since gethostbyname() and
getaddrinfo() seem to return strings:
$ ./python -m timeit -s import socket; s =
socket.socket(socket.AF_INET, socket.SOCK_DGRAM);
addr=socket.gethostbyname('127.0.0.1') s.sendto(b'hello', (addr,
4242))
$ ./python -c import socket; print(type(socket.gethostbyname('127.0.0.1')))
class 'str'
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22127
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22120] Fix compiler warnings
Charles-François Natali added the comment: This patch should probably be moved to its own issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22120 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22120] Fix compiler warnings
Changes by Charles-François Natali cf.nat...@gmail.com: -- Removed message: http://bugs.python.org/msg224550 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22120 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22110] enable extra compilation warnings
Charles-François Natali added the comment: Committed. Sorry for the extra ~70 warnings :-) -- resolution: - fixed stage: - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22110 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22110] enable extra compilation warnings
Charles-François Natali added the comment: Antoine Pitrou added the comment: Enabling the warnings may be a good incitation for other people to fix them ;) That was my intention... Can I push it, and let warnings be fixed on a case-by-case basis? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22110 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19923] OSError: [Errno 512] Unknown error 512 in test_multiprocessing
Charles-François Natali added the comment: Closing, since it's likely a kernel bug. -- resolution: - third party stage: - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19923 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15152] test_subprocess failures on awfully slow builtbots
Charles-François Natali added the comment: Closing, I haven't seen this in a while. -- resolution: - out of date stage: needs patch - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15152 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17085] test_socket crashes the whole test suite
Charles-François Natali added the comment: Thanks for the reminder Mark. Yes, it is probably still an issue with the latest 2.7 release. There were actually two issues: - send send()/sendall() call didn't block because the test doesn't write enough data: we have since added a SOCK_MAX_SIZE constant to test_support just for that purpose, and the test has been updated, so it should now block (and succeed). - the crash occurs because the test doesn't reset the alarm (with alarm(0)) upon exit, so if the alarm didn't go off during the test, it comes later when the original signal handler has been reset, and the default handler just exits. To sum up, the first point is already fixed, and for the second point, here's a patch. Note that 3.4 and default aren't affected, because the alarm is properly reset. -- keywords: +patch nosy: +neologix Added file: http://bugs.python.org/file36182/reset_sigalrm_handler.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17085 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22110] enable extra compilation warnings
New submission from Charles-François Natali: The patch attached enables -Wsign-compare and -Wunreachable-code if supported by the compiler. AFAICT, mixed sign comparison warning is automatically enabled by Microsoft's compiler, and is usually a good thing. It does add some warnings though. As for unreachable code, it's also usually a good thing, since it can be a source of bugs. Note that it's not enabled in debug mode, since in debug mode the code paths aren't the same. -- components: Build files: extra_warnings.diff keywords: patch messages: 224349 nosy: haypo, neologix, pitrou priority: normal severity: normal status: open title: enable extra compilation warnings type: enhancement versions: Python 3.5 Added file: http://bugs.python.org/file36169/extra_warnings.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22110 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18174] Make regrtest with --huntrleaks check for fd leaks
Charles-François Natali added the comment: Richard Oudkerk added the comment: I can't remember why I did not use fstat() -- probably it did not occur to me. I probably have Alzeihmer, I was sure I heard a reasonable case for dup() vs fstat(). The only thing I can think of is that fstat() can incur I/O, whereas dup() shouldn't. In any case, I too prefer fstat(), since it doesn't require creating a new FD (which could fail with EMFILE/ENFILE), and is more logical. The only comment I have about this patch is I think that the helper function to detect the number of open FD might have its place in test.support. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18174 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22054] Add os.get_blocking() and os.set_blocking() functions
Charles-François Natali added the comment: I agree with Akira, although it's probably too late now to rename. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22054 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19875] test_getsockaddrarg occasional failure
Charles-François Natali added the comment: Backported to 2.7 (don't know how Iforgot it). 3.3 is only open for security issues, so not backporting. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19875 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22018] Add a new signal.set_wakeup_socket() function
Charles-François Natali added the comment: Pipes cannot be configured in non-blocking mode on Windows. It sounds dangerous to call a blocking syscall in a signal handler. In fact, it works to write the signal number into a pipe on Windows, but I'm worried about the blocking behaviour. OK, but if someone passes a socket in blocking mode, it will be accepted. I don't understand why, if you're worried about a blocking write, you don't just check that the FD passed is in non-blocking mode. It's conceptually cleaner, more direct and safer. Also, I think there's another issue open on the tracker to check just that, so I'd leave this check out here. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22018] Add a new signal.set_wakeup_socket() function
Charles-François Natali added the comment: In the issue #22042, I would like to make automatically the file desscriptor or socket handler in non-blocking mode. The problem is that you cannot make a file descriptor in non-blocking mode on Windows. I don't think we should set it non-blocking automatically, but rather check that it's non-blocking. The first reason I can think of is that the user passing a blocking FD could be a sign of a bug (e.g. if the other end is in blocking mode too), and we shouldn't silently work-around it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22018 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
