Charles-François Natali added the comment:
The patch is not compatible with 3.4. Does this bug exist in 3.4?
No, selectors all expect a timeout in seconds, so this should be fixed in 3.4.
--
___
Python tracker rep...@bugs.python.org
http
Charles-François Natali added the comment:
The test is failing on Windows buildbot:
http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%203.x/builds/1851/steps/test/logs/stdio
==
ERROR
New submission from Charles-François Natali:
This is a spinoff of issue #19506: currently, subprocess.communicate() uses a
4K buffer when reading data from pipes.
This was probably optimal a couple years ago, but nowadays most operating
systems have larger pipes (e.g. Linux has 64K), so we
Charles-François Natali added the comment:
STINNER Victor added the comment:
I don't like generic except OSError: pass. Here is a first patch for
epoll() to use except FileNotFoundError: pass instead. Kqueue selector
should also be patched.
Except that it can fail with ENOENT, but also
Charles-François Natali added the comment:
STINNER Victor added the comment:
Since Popen.communicate() returns the whole content of the buffer, would it
be safe to increase the buffer size? For example, use 4 GB as the buffer size?
Sure, if you want to pay the CPU and memory overhead
Charles-François Natali added the comment:
Looks like a kernel bug.
errno 512 is ERESTARTSYS, which shouldn't leak to user-mode.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19923
Charles-François Natali added the comment:
Roundup Robot added the comment:
New changeset 03a056c3b88e by Gregory P. Smith in branch '3.3':
Fixes issue #19929: Call os.read with 32768 within subprocess.Popen
http://hg.python.org/cpython/rev/03a056c3b88e
Not that it bothers me, but AFAICT
Charles-François Natali added the comment:
I will look into the Windows problem, but I suspect the best we can do there
is skip the test.
I already took care of that:
http://hg.python.org/cpython/rev/01676a4c16ff
--
___
Python tracker rep
Charles-François Natali added the comment:
LGTM!
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19876
___
___
Python-bugs-list mailing list
Changes by Charles-François Natali cf.nat...@gmail.com:
--
resolution: - fixed
stage: needs patch - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19857
Charles-François Natali added the comment:
Sorry for the delay, I didn't have any free time this week.
I'll review the patch shortly, but the idea sounds fine (I just need
to check if we can't be a little more specific for errnos upon
unregister
Charles-François Natali added the comment:
Guido van Rossum added the comment:
Here's an attempt at fixing the ValueError.
I don't like the exhaustive search much, but the alternative is to maintain
an inverse dict. What do you think?
I was going to suggest such an exhaustive search.
I
Charles-François Natali added the comment:
Thanks!
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19850
Charles-François Natali added the comment:
I agree with Antoine.
Silencing warning is fine, as long as it's not to the detriment of
clarity (see Debian Openssl's vulnerability extreme example).
--
___
Python tracker rep...@bugs.python.org
http
Charles-François Natali added the comment:
This test is inherently subject to a race condition:
port = support.find_unused_port()
[...]
try:
self.assertRaises(OverflowError, sock.bind, (host, big_port))
self.assertRaises(OverflowError, sock.bind
Charles-François Natali added the comment:
Well, unregister() documentation currently contains this:
.. method:: unregister(fileobj)
Unregister a file object from selection, removing it from monitoring. A
file object shall be unregistered prior to being closed.
I'm not sure
Charles-François Natali added the comment:
The patch is fine, but it is hard to rely on it to prevent bugs from
happening because that requires cooperation from all modules registering
signal handlers.
Once again, that's why the bug report says *limit* EINTR occurrences.
We all know
Charles-François Natali added the comment:
Guido van Rossum added the comment:
I think you're looking for the discussion in issue 19017.
IIRC the conclusion is that not only do you not get the same error
everywhere, but you get it at different points -- sometimes register() of a
bad FD
Charles-François Natali added the comment:
STINNER Victor added the comment:
Kristján Only that issue #10517 mentions reasons to keep the old behavior,
specifically http://bugs.python.org/issue10517#msg134573 (...)
@Kristján: The behaviour of PyThread_set_key() discussed in this issue
Charles-François Natali added the comment:
Guido van Rossum added the comment:
That's just rhetoric -- I am beginning to believe that nobody has any data.
Here's some opposite rhetoric. If it ain't broke, don't fix it. Plus, if
it's so much better, why isn't it the default? If you say
Charles-François Natali added the comment:
Guido van Rossum added the comment:
LGTM, although I wish that you had time to implement the comment TODO:
Subclasses can probably optimize this even further rather than just
shuffling the existing code around. :-)
I will look a this, but I
Charles-François Natali added the comment:
Just for the record, I was initially in favor of recomputing the
timeout and retrying upon EINTR, but Guido prefers to return empty
lists, and since that's a better compromise than the current situation
(I've seen *many* people complaining on EINTR
Charles-François Natali added the comment:
STINNER Victor added the comment:
I think that's a cleaner design.
For common containers like Sequence or Mapping, ABC are useful. But I don't
expect new implementations of BaseSelector. Is it just for purity? :-p
There are already
Charles-François Natali added the comment:
Impressive speedup :-)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19506
___
___
Python-bugs-list
Charles-François Natali added the comment:
Well this is wishing thinking, since by returning an empty list you
force the user to handle EINTR - just in a different way.
I know that returning an empty list changes the semantics: I just
think that's better - or not as bad - than the current
New submission from Charles-François Natali:
asyncio makes heavy use of SIGCHLD for subprocesses.
A consequence of this if that a lot of syscalls can fail with EINTR (see e.g.
issue #18885).
The attached patch uses SA_RESTART (through signal.siginterrupt()) to limit
EINTR occurrences, e.g
Charles-François Natali added the comment:
How about this?
--
keywords: +needs review, patch
stage: - patch review
Added file: http://bugs.python.org/file32924/selectors_select_signal.diff
___
Python tracker rep...@bugs.python.org
http
Changes by Charles-François Natali cf.nat...@gmail.com:
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
versions: +Python 3.2 -Python 3.4
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Changes by Charles-François Natali cf.nat...@gmail.com:
--
status: open - pending
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18843
Changes by Charles-François Natali cf.nat...@gmail.com:
--
resolution: - rejected
stage: patch review - committed/rejected
status: pending - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1611154
Charles-François Natali added the comment:
Vajrasky, thanks for the patch!
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18994
Charles-François Natali added the comment:
Thanks for the feedback!
--
resolution: - invalid
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18843
Charles-François Natali added the comment:
It sounds like doing this is fine (as Glyph suggests in the email thread) but
it isn't magically going to solve all EINTR problems, just reduce the number
of calls that could encounter them.
Indeed, hence *limit* EINTR occurrences :-)
Note
New submission from Charles-François Natali:
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/5901/steps/test/logs/stdio
[134/387/1] test_signal
Timeout (1:00:00)!
Thread 0x000801e24800 (most recent call first):
File
/usr/home/buildbot/buildarea/3.x.krah
Charles-François Natali added the comment:
Could someone with a dual-core machine try the attached simplistic
benchmark with and without Victor's patch?
I can see some user-time difference with 'time' on my single-core
machine, but I'm curious to see how this would affect things were both
New submission from Charles-François Natali:
Initially, BaseSelector was simply designed as the base implementation used by
concrete ones like SelectSelector Co.
Then BaseSelector evolved to be an ABC, but the problem is that it's really not
usable as such: the register() and unregister
Charles-François Natali added the comment:
Alright, here's a first step: select/poll/epoll/etc now return empty
lists/tuples upon EINTR. This comes with tests (note that all those tests
could probably be factored, but that's another story).
--
keywords: +needs review, patch
stage: needs
Charles-François Natali added the comment:
I'm wondering, is there a reason we made BaseSelector a public API?
The idead was to have an ABC so that users can implement their own
selector, and pass it to e.g. asyncio or anything alse expecting a
selector.
Other than that, the only use
Charles-François Natali added the comment:
The problem for documentation use is that we're christening it as an
official API, and thus it becomes more difficult to refactor the
inheritance hierarchy.
So what would you suggest?
--
___
Python
Charles-François Natali added the comment:
Here's a patch adding socketpair to test.support.
This version has been used in test_selectors for quite some time now,
and would probably be useful for other tests as well.
--
Added file: http://bugs.python.org/file32909/socketpair-1.diff
New submission from Charles-François Natali:
Here's a patch improving the examples section for the selectors module.
--
assignee: docs@python
components: Documentation
files: selectors_examples.diff
keywords: needs review, patch
messages: 204750
nosy: docs@python, neologix
priority
Changes by Charles-François Natali cf.nat...@gmail.com:
Removed file: http://bugs.python.org/file32894/selectors_examples.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19836
Changes by Charles-François Natali cf.nat...@gmail.com:
Added file: http://bugs.python.org/file32895/selectors_examples.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19836
Charles-François Natali added the comment:
Here's an updated patch adding your echo server to the examples section.
--
Added file: http://bugs.python.org/file32899/selectors_example-1.diff
___
Python tracker rep...@bugs.python.org
http
Charles-François Natali added the comment:
Antoine Pitrou added the comment:
Calling it _PyThread_set_key_value is prone to confusion.
Ideally we would fix PyThread_set_key_value's behaviour. Are there users of
the function outside from CPython?
The question is why does it behave this way
Charles-François Natali added the comment:
I'm with Antoine, this is *really* going too far.
SOCK_CLOEXEC and friends are useful to avoid race conditions: there's
no such concern with the non-blocking flag.
Shaving one or two syscalls is IMO completely useless, and doesn't
justify the extra code
Charles-François Natali added the comment:
AFAICT, there's no link (FWIW I wrote the patch for #10517, then the fix
for threads created outside of Python calling into a subinterpreter -
issue #13156).
Actually, this fix would have avoided the regression in issue #13156.
But since it's tricky
Charles-François Natali added the comment:
STINNER Victor added the comment:
@neologix: I would like your opinion first, because you accepted the PEP.
Well, I'm not a native speaker, but peak does sound better than
max (I'd say the later refers to an externally-enforced limit,
contrarily
Charles-François Natali added the comment:
IMO, this functionality is subsumed by http://bugs.python.org/issue19776
(Path.expanduser).
--
nosy: +neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19777
Charles-François Natali added the comment:
Antoine Pitrou added the comment:
I think this is more of a documentation issue. People who don't want a new fd
can hardcode PollSelector (poll has been POSIX for a long time).
That's also what I now think.
I don't think that the use case
Charles-François Natali added the comment:
Victor, is the attached patch up-to-date?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
Charles-François Natali added the comment:
If you really want to use packing, keep it.
But please remove this:
+/* ensure that the frame_t structure is packed */
+assert(sizeof(frame_t) == (sizeof(PyObject*) + sizeof(int)));
--
___
Python
Charles-François Natali added the comment:
STINNER Victor added the comment:
I'm not convinced myself that hashtable.c/h can be reused immediatly, so I
prefer to move these two files to Modules. The files may be moved later if
the containers are reused.
Please do so. I'd also like
Charles-François Natali added the comment:
Serhiy Storchaka added the comment:
I don't know. But a function call have a cost.
I'm with Serhiy here.
Writing a performance optimization without benchmark is generally a
bad idea (I won't quote Knuth
Charles-François Natali added the comment:
I made a review at http://bugs.python.org/review/18874/#ps9860 (not sure you
got notified).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
Charles-François Natali added the comment:
Would it be possible to generate a clean patch?
The latest one contains many unrelated commits.
--
nosy: +neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
Charles-François Natali added the comment:
mpb added the comment:
Someone wrote a kernel patch based on my bug report.
http://www.spinics.net/lists/netdev/msg257653.html
It's just a patch to avoid returning garbage in the address.
But AFAICT, recvfrom() returning 0 is enough to know
Charles-François Natali added the comment:
R. David Murray added the comment:
neologix noted that *when redirection is used* the way that *all* windows
file handles are inherited changes.
That's true (but that was from Richard actually
Charles-François Natali added the comment:
I think it's simply due to file descriptor inheritance (files being
inherited by other subprocess instance): since Windows can't remove open
files, kaboom. It doesn't have anything to do with threads.
--
nosy: +neologix
Charles-François Natali added the comment:
Bernt Røskar Brenna added the comment:
@neologix: How can it not have anything to do with threads?
- It always works with max_workers == 1
- When max_workers == 2, shutil.rmtree sometimes fails
It has nothing to do with threads.
You could
Charles-François Natali added the comment:
After some research...
Which is normal, since UDP sockets aren't connected.
But UDP sockets can be connected!
No, they can't.
Connecting a UDP socket doesn't established a duplex connection like
in TCP: it's just a shortand for not having
Changes by Charles-François Natali cf.nat...@gmail.com:
--
resolution: - wont fix
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18907
Changes by Charles-François Natali cf.nat...@gmail.com:
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18923
Charles-François Natali added the comment:
When I try this with a UDP socket, the thread calling shutdown
raises an OS Error (transport end point not connected).
Which is normal, since UDP sockets aren't connected.
In contrast, sock.close does not cause the blocked thread to unblock
Changes by Charles-François Natali cf.nat...@gmail.com:
Added file: http://bugs.python.org/file32549/3bf73dcd0b42.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
Charles-François Natali added the comment:
Here's an updated patch with a better logic: in the previous version - based on
current poll-based implementation, the FD was inferred from the event (i.e.
read ready - stdout/stderr, write ready - stderr). The new version directly
checks the ready
Charles-François Natali added the comment:
What is the status of this issue?
AFAICT, we haven't reached a consensus yet on the best way to handle EBADF.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19017
Charles-François Natali added the comment:
By which you mean that you still don't agree with my proposal? Which is to
fix it for most syscalls but not for select(), poll() and similar (anything
that the new selectors module interfaces to).
I agree with your proposal, but that's another issue
Changes by Charles-François Natali cf.nat...@gmail.com:
Added file: http://bugs.python.org/file32448/subprocess_selectors-2.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18923
Charles-François Natali added the comment:
There are actually two reasons to choosing poll over epoll/kqueue
(i.e. no extra FD):
- it's a bit faster (1 syscall vs 3)
- but more importantly - and that's the main reason I did it in
telnetlib/multiprocessing/subprocess - sometimes, you really don't
Charles-François Natali added the comment:
Given PEP 446 (fds are now CLOEXEC by default) I prepared an updated patch
where the fork lock is undocumented and subprocess no longer uses the fork
lock. (I did not want to encourage the mixing of threads with fork() without
exec() by exposing
Charles-François Natali added the comment:
Of course, when I have 300 connections to remote nodes, I use poll()
to multiplex between them.
But there are times when you can have a large number of threads
running concurrently, and if many of them call e.g.
subprocess.check_output() at the same
Charles-François Natali added the comment:
Berker Peksag added the comment:
+ .. method:: get_map()
+
+ Return a mapping of file objects to selector keys.
I kind of feel like a walking linter though I think this needs a versionadded
tag :)
Well, selectors have been added to 3.4
Charles-François Natali added the comment:
I agree this should be fixed.
Robert, want to submit a patch?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19444
Charles-François Natali added the comment:
Here's an updated patch using the new selector.get_map() method.
It removes ~100 lines to subprocess, which is always nice.
--
Added file: http://bugs.python.org/file32429/subprocess_selectors-1.diff
Charles-François Natali added the comment:
Committed.
Antoine, thanks for the idea and patch!
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Charles-François Natali added the comment:
Here's another failure:
==
FAIL: test_interpreter_shutdown
(test.test_concurrent_futures.ProcessPoolShutdownTest
Charles-François Natali added the comment:
Also, unless people haven't been reporting other errors, these buildbots have
been stable up until recently
Well, the fact that several tests - which were working just fine -
started failing randomly recently (like test_concurrent_futures
Charles-François Natali added the comment:
Seriously, nobody gives a ... about SPARC and MIPS. :) It's nice that
Python still works on these CPU architectures. But I neither want to
deviate from the SipHash24 implementation nor make the code slower on
all relevant platforms such as X86
Charles-François Natali added the comment:
Well, unaligned memory access is usually slower on all architectures :-)
Also, I think some ARM architectures don't support unaligned access, so
it's not really a thing of the past...
On modern computers it's either not slower or just a tiny bit
Charles-François Natali added the comment:
Note that no matter where the atfork-hook is executed (prepare,
parent, child), it will still be deadlock-prone in case of
double-fork, since the hook will effectively be called after the first
fork (and before the second one). And double-fork is common
Charles-François Natali added the comment:
I didn't expect to see a combination of glibc with prlimit() and Kernel
without prlimit(). According to man prlimit it's not suppose to fail with
ENOSYS, too.
Yeah, we've seen this several times on some buildbots.
Note that actually, any syscall
Charles-François Natali added the comment:
This failure doesn't make sense: whether SIGALRM is delivered or not, epoll()
should time out after 2 seconds, not 105s.
What's more, here's another test_subprocess failure:
==
ERROR
Charles-François Natali added the comment:
Flaky VM clock perhaps?
Yeah, thought about that, but here we see *exactly* the same value for
two time intervals (using time.monotonic()), computed over different
test suite runs. I'd expected something more erratic
Charles-François Natali added the comment:
Could you try to run the test in a loop under strace (strace -ttTf) ?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19386
Charles-François Natali added the comment:
Victor, did you see Serhiy's comment?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19339
Charles-François Natali added the comment:
Antoine Pitrou added the comment:
FWIW, I think the ideal solution would be for keys() (*) to return a
read-only Mapping implementation, allowing for file object lookup (using
__getitem__) as well as iteration on SelectorKeys (using __iter__
Charles-François Natali added the comment:
The test is failing with ENOSYS on one of the buildbots:
==
ERROR: test_prlimit (test.test_resource.ResourceTest
Charles-François Natali added the comment:
I'm also skeptical: I doubt the list allocation weights much compared
to the syscall.
And like Christian, I'm curious to know which implementation you'd chose.
--
___
Python tracker rep...@bugs.python.org
Charles-François Natali added the comment:
I'm completely lost: how many issues is this thread conflating?
For example, David, I think you said that my patch regarding signal
handling did fix some tests.
Is that still the case?
If yes, could you open a specific issue for the remaining failing
Charles-François Natali added the comment:
David Edelsohn added the comment:
This enhancement issue is not a demand that you or anyone else in the Python
community volunteer to implement it. AIX is a very lucrative business and
deployed in many businesses that the Python community would
New submission from Charles-François Natali:
(This is a spinoff from http://bugs.python.org/issue19293#msg200598)
When SIGCHLD is received, _sig_chld() is executed:
def _sig_chld(self):
[...]
transp = self._subprocesses.get(pid)
if transp is not None
Charles-François Natali added the comment:
Just to let you know that I'm not forgetting this issue.
I'd just like to iron-out issue #19172 (keys()-like method for selectors),
since it could impact the API, to avoid repeated refactoring
Charles-François Natali added the comment:
Gregory, did you make any progress on this?
I think it would be a nice addition.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11406
Charles-François Natali added the comment:
Christian Heimes added the comment:
I think it's more likely that my patch is triggering an existing bug. The
locking code for the SSL module and OpenSSL doesn't release locks on fork. I
have attached an experimental patch that unlocks all locks
Charles-François Natali added the comment:
I took the freedom to push a fix for the test_pydoc failures (all the buildbots
were red). It should fix the failures, but since I'm not familiar with the
code, it'd be good to have someone double-check it.
Ethan, you just broke all the buildbots
Charles-François Natali added the comment:
I have a couple random remarks:
- now that FDs are non-inheritable by default, fork locks around
subprocess and multiprocessing shouldn't be necessary anymore? What
other use cases does the fork-lock have?
- the current implementation keeps hard
Charles-François Natali added the comment:
Other parts of your reasoning are somewhat circular because most of the
problems on AIX are Linux-isms (or BSD-isms where BSD has not yielded to
Linux behavior). So you argue against adding lots of AIX-specific code, yet
fixing some of the tests
Charles-François Natali added the comment:
Richard Oudkerk added the comment:
- now that FDs are non-inheritable by default, fork locks around
subprocess and multiprocessing shouldn't be necessary anymore? What
other use cases does the fork-lock have?
CLOEXEC fds will still be inherited
Charles-François Natali added the comment:
This is by design. Let me try to defend the design.
OK, if that's a know limitation, then that's fine.
It would be nice to add maybe a note somewhere in the documentation, so that
people don't get bitten by this (and also probably add a test
301 - 400 of 1861 matches
Mail list logo