[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: Added file: http://bugs.python.org/file23684/11f08326afd0.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment:
New changeset. The only change is:
diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c
--- a/Modules/selectmodule.c
+++ b/Modules/selectmodule.c
@@ -685,8 +685,16 @@
return -1;
}
if (n size) {
-PyErr_SetString(PyExc_IOError, failed to write all pollfds.
-Please, report in http://bugs.python.org/;);
+/*
+** Data writed to /dev/poll is a binary data structure. It is not
+** clear what to do if a partial write occurred. For now, raise
+** an exception and see if we actually found this problem in
+** the wild.
+*/
+PyErr_Format(PyExc_IOError, failed to write all pollfds.
+Please, report at http://bugs.python.org/.
+Data to report: Size tried: %d, actual size written: %d,
+size, n);
return -1;
}
return 0;
If there are no disagreements, I will commit to default (3.3) soon (in a few
hours/a day).
Thanks!.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6397
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 8f7ab4bf7ad9 by Jesus Cea in branch 'default': Issue #6397: Support '/dev/poll' polling objects in select module, under Solaris derivatives. http://hg.python.org/cpython/rev/8f7ab4bf7ad9 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: Added file: http://bugs.python.org/file23643/2506e49b9f71.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment: Another changeset. Hopefully, final :-). Please, review. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Ross Lagerwall rosslagerw...@gmail.com added the comment: + increases this value, c:func:`devpoll` will return a possible + incomplete list of active file descriptors. I think this should change to: + increases this value, c:func:`devpoll` will return a possibly + incomplete list of active file descriptors. or even better: + increases this value, c:func:`devpoll` may return an + incomplete list of active file descriptors. Cheers -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment: Thanks, Ross. Your suggestion has been committed in my branch. Waiting for more feedback. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Ross Lagerwall rosslagerw...@gmail.com added the comment: Is write()ing a devpoll fd a blocking operation in the kernel? Does it need to have Py_BEGIN_ALLOW_THREADS around it? The same question applies for open()ing it. Obviously, the ioctl() call *is* blocking :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Ross Lagerwall rosslagerw...@gmail.com added the comment: Also, you can use Py_RETURN_NONE instead of: +Py_INCREF(Py_None); +return Py_None; -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment:
They are operations potentially slow, there are not realtime specifications.
My machine is quite capable (16 CPUs), but let's see what a bit of DTRACE
scripting tells us:
First, let's time the open:
syscall::open*:entry
/copyinstr(arg0)==/dev/poll/
{
self-ts = timestamp;
}
syscall::open*:return
/self-ts/
{
@stats = quantize(timestamp-self-ts);
self-ts = 0;
}
The result, times in nanoseconds:
value - Distribution - count
2048 | 0
4096 | 4497743
8192 | 17358
16384 | 1592
32768 | 2046
65536 | 812
131072 | 374
262144 | 0
Most opens finish in 4 microseconds, but we have a handful of slow openings
of hundred of microseconds.
Anyway, argueing with the GIL here is a nonissue, since the open is only done
at object creation, and a sane program will only create a handful of devpoll
objets.
Let's see now the write:
syscall::open*:entry
/copyinstr(arg0)==/dev/poll/
{
self-ts = timestamp;
}
syscall::open*:return
/self-ts/
{
@stats = quantize(timestamp-self-ts);
self-ts = 0;
}
The results for a single descriptor registered:
value - Distribution - count
256 | 0
512 | 4102701
1024 | 28514
2048 | 11269
4096 | 537
8192 | 245
16384 | 377
32768 | 193
65536 | 134
131072 | 71
262144 | 0
Most writes are really fast, half a microsecond, but we have sporadic latencies
of a hundred of microseconds.
Re-registering 200 sockets per loop, I have:
value - Distribution - count
512 | 0
1024 | 50
2048 | 94
4096 | 157
8192 |@@@ 1112314
16384 |@22977
32768 | 1584
65536 | 842
131072 | 168
262144 | 0
Most writes takes around 8 microseconds.
So now the problem is to estimate how much time I need to release readquire
the GIL. For that, and while we don't have DTRACE integration in python (my
next project, I hope), I change sourcecode to add a manual probe using the
Solaris High Resolution timers, I get a median time of around 550 nanoseconds
(half a microsecond), with a few spikes, when the GIL is not contested.
So, unlocking the GIL is adding around half microsecond to the syscalls. This
is fast, but comparable with the syscall actual duration. Liberating the GIL is
doubling the time to registerunregister a socket (supposely we are doing a
single socket per iteration, something realistic in real code).
Being realistic, any work we do with the descriptors (like actually reading or
writing it) is going to make this 0.5 microsecond gain pointless. Freeing the
GIL, we are protected of any kernel contention, and playing for the rules :-).
Anyway I am open to feedback.
PS: I also checked with GIL contention, and results are comparable. That is
surprising, maybe related to the GIL improvements in 3.3. I haven't
investigated the issue further.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6397
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: Added file: http://bugs.python.org/file23646/9d687fdd924d.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment: New changeset, after Ross feedback. Thanks!. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Ross Lagerwall rosslagerw...@gmail.com added the comment:
That was thorough :-) Seems OK though.
+if (n size) {
+PyErr_SetString(PyExc_IOError, failed to write all pollfds.
+Please, report in http://bugs.python.org/;);
If n size, it's not a Python error is it? I would say it's the OS's fault.
Otherwise, it looks good... although I don't have currently access to a Solaris
box to test it.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6397
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment: The timing for the GIL I am providing is for releaseacquiring. That is, all the work. In fact I am having in count too the syscall inside the releaseacquire, to account for cache issues. That is, between the release and the acquire, there is a real syscall. Of course, I am not timing it. So, the timing I am providing is accurate. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Antoine Pitrou pit...@free.fr added the comment:
That was thorough :-) Seems OK though.
+if (n size) {
+PyErr_SetString(PyExc_IOError, failed to write all pollfds.
+Please, report in http://bugs.python.org/;);
If n size, it's not a Python error is it? I would say it's the OS's fault.
No, but it's a Python error if Python wrongly assumes that write() won't
return a partial result. Ideally write() should be retried in a loop
until everything is written out.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6397
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment: The problem with partial writes is that the data is not an unstructured stream of bytes, but a concrete binary dump. You can not simply retry again. My bet is that /dev/poll neves does partial writes. If I am mistaken, I will bug the Illumos people to help me to solve it. So far, this seems a theorical problema. Just in case it is not, we raise an exception and urge the programmer to report us the issue. The other option would be to try to be clever and do things like retrying if the partial write is a multiple of the struct size, but what if it not so?. Instead of guessing and play games, I rather prefer to know if this is actually a problem in the wild. In my tests, with 65530 sockets, I never saw this impossible exception. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Giampaolo Rodola' g.rod...@gmail.com: -- nosy: +giampaolo.rodola ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: Added file: http://bugs.python.org/file23635/c47a846bed1a.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: Removed file: http://bugs.python.org/file23635/c47a846bed1a.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: Added file: http://bugs.python.org/file23636/518b32ce893e.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment: Please, check the new changeset, with all your feedback. Thanks!. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: Removed file: http://bugs.python.org/file23532/0b701eb5e9e3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: Added file: http://bugs.python.org/file23626/528fdd816160.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Jesús Cea Avión j...@jcea.es added the comment: Please, review. With current code, each devpoll object has capacity for managing 256 fds, by default. This is about 2048 bytes. The cost seems reasonable, since a normal program will have only a few devpoll objects around. I have considered an optional parameter to tune this, but interaction with rlimit is messy. Even if we manage 65536 fds, the memory cost is about 512Kbytes per devpoll, and you surely can affort it if you are actually managing 65536 descriptors... The code is not threadsafe. It doesn't crash, but concurrent use of a devpoll has undefined results. Please, review for integration. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Ross Lagerwall rosslagerw...@gmail.com: -- nosy: +rosslagerwall ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6397] Implementing Solaris /dev/poll in the select module
Changes by Jesús Cea Avión j...@jcea.es: -- title: Implementing Solaris poll in the select module - Implementing Solaris /dev/poll in the select module ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6397 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
