Richard Oudkerk added the comment:
It seems that the return code of WSAPoll() does not include the count of array
items with revents == POLLNVAL. In the case where all of them are POLLNVAL,
instead of returning 0 (which usually indicates a timeout) it returns -1 and
WSAGetLastError
Changes by Richard Oudkerk shibt...@gmail.com:
--
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16616
___
___
Python-bugs
Changes by Richard Oudkerk shibt...@gmail.com:
--
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15526
___
___
Python-bugs
Richard Oudkerk added the comment:
Here is a new version with tests and docs.
Note that the docs do not mention the bug mentioned in
http://daniel.haxx.se/blog/2012/10/10/wsapoll-is-broken/
Maybe they should?
Note that that bug makes it a bit difficult to use poll with tulip on Windows
Richard Oudkerk added the comment:
When you run wy.py the wow module gets partially imported, and then garbage
collected because it fails to import successfully. The destructor for the
module replaces values in the module's __dict__ with None. So when the cleanup
function runs you get
Richard Oudkerk added the comment:
Things should be better in the future, when modules are cleared with true
garbage collection.
When is this future of which you speak?
I am not sure whether it would affect performance, but a weakrefable subclass
of dict could be used for module dicts
Richard Oudkerk added the comment:
See issue812369 for the shutdown procedure and modules cleanup.
I am aware of that issue, but the original patch is 9 years old. Which is why
I ask if/when it will actually happen.
--
___
Python tracker rep
New submission from Richard Oudkerk:
Relevant code:
int timeout = 0, poll_result, i, j;
...
tout = PyNumber_Long(tout);
if (!tout)
return NULL;
timeout = PyLong_AsLong(tout); -- implicit cast to int
--
messages: 177811
nosy: sbt
priority
Richard Oudkerk added the comment:
Thanks. I will close.
--
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16736
Richard Oudkerk added the comment:
Perhaps the simplest thing would be to stop doing anything special when a
module is garbage collected: the garbage collector can take care of any
orphaned ref-cycles involving the module dict. Then at shutdown the remaining
modules in sys.modules could have
Richard Oudkerk added the comment:
I suspect that the size of the 5GB file is originally a 64 bit quantity, but
gets cast unsafely to a 32 bit size_t to give 1GB. This is causing the
miscalculations.
There is no way to map all of a 5GB file in a 32 bit process -- 4GB is the
maximum -- so
Richard Oudkerk added the comment:
This bit looks wrong to me:
if (offset - size PY_SSIZE_T_MAX)
/* Map area too large to fit in memory */
m_obj-size = (Py_ssize_t) -1;
Should it not be size - offset instead of offset - size? (offset and size
Richard Oudkerk added the comment:
On 32 bit Unix mmap() will raise ValueError(mmap length is too large) in
Marc's example. This is correct since Python's sequence protocol does not
support indexes larger than sys.maxsize.
But on 32 bit Windows, if length == 0 then the size check always
Richard Oudkerk added the comment:
This change is not backward compatible. Now user can mmap a larger file
and safely access lower 2 GiB. With the patch it will fail.
They should specify length=2GiB-1 if that is what they want.
With length=0 you can only access the lower 2GiB if file_size
Richard Oudkerk added the comment:
New patch with same check for Unix.
--
Added file: http://bugs.python.org/file28446/mmap.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16743
Richard Oudkerk added the comment:
Isn't 2 GiB + 1 bytes mmap file enough for testing?
Yes.
But creating multigigabyte files is very slow on Windows. On Linux/FreeBSD
test_mmap takes a fraction of a second, whereas on Windows it takes over 2
minutes. (Presumably Linux/FreeBSD
Changes by Richard Oudkerk shibt...@gmail.com:
--
title: mmap accepts files 1 GB, but processes only 1 GB - mmap on Windows
can mishandle files larger than sys.maxsize
type: enhancement - behavior
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
Richard, apart from performance, what's the advantage of this approach over
the
fork+exec version?
It is really just performance. For context running the unittests in a 1 cpu
linux VM gives me
fork:
real0m53.868s
user0m1.496s
sys 0m9.757s
Richard Oudkerk added the comment:
Numbers when running on Linux on a laptop with 2 cores + hyperthreading.
RUNNING UNITTESTS:
fork:
real0m50.687s
user0m9.213s
sys 0m4.012s
fork+exec:
real1m9.062s
user0m48.579s
sys 0m6.648s
forkserver
Changes by Richard Oudkerk shibt...@gmail.com:
Added file: http://bugs.python.org/file28461/8f08d83264a0.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8713
Richard Oudkerk added the comment:
The safest default would be fork+exec though we need to implement the
fork+exec code as a C extension module or have it use subprocess (as I
noted in the mb_fork_exec.patch review).
That was an old version of the patch.
In the branch
http
New submission from Richard Oudkerk:
The actual signature is
socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None)
but the documented signature is
socket.socket([family[, type[, proto]]])
Should the fileno argument be documented or is it considered an implementation
Richard Oudkerk added the comment:
The fileno argument looks like an implementation detail to me.
It has at least one potential use. On Windows socket.detach() returns a socket
handle but there is no documented way to close it -- os.close() will not work.
The only way to close it that I
Richard Oudkerk added the comment:
There is an alternative (documented) interface:
socket.fromfd(handle, socket.AF_INET, socket.SOCK_STREAM).close()
socket.fromfd() duplicates the handle, so that does not close the original
handle
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9586
Richard Oudkerk added the comment:
Note that on Windows there is an O_NOINHERIT flag which almost corresponds to
O_CLOEXEC on Linux.
I don't think there is a need to use the win32 api.
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
A while ago I did write a PipeIO class which subclasses io.RawIOBase and works
for overlapped pipe handles. (It was intended for multiprocessing and doing
asynchronous IO with subprocess.)
As it is it would not work with normal files because when you do
Richard Oudkerk added the comment:
Attached is a module for Python 3.3+ which subclasses io.RawIOBase. The
constructor signature is
WinFileIO(handle, mode=r, closehandle=True)
where mode is r, w, r+ or w+. Handles can be created using
_winapi.CreateFile().
Issues:
- No support
Changes by Richard Oudkerk shibt...@gmail.com:
Added file: http://bugs.python.org/file28545/test_winfileio.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Richard Oudkerk added the comment:
Is this actually a problem?
If events are arranged in a queue and epoll_wait() just removes the oldest
events (up to maxevents) from that queue then there would be no problem with
using a small value for maxevents.
I don't *know* if that is the case, but I
Richard Oudkerk added the comment:
Attached is a patch which adds a winio module which is a replacement for io,
but uses windows handles instead of fds.
It reimplements FileIO and open(), and provides openhandle() and closehandle()
as replacements for os.open() and os.close().
test_io has
Richard Oudkerk added the comment:
I actually wrote a script to reproduce this issue:
The program does *not* demonstrate starvation because you are servicing the
resource represented by the starved duplicate fds before calling poll() again.
You are creating thousands of duplicate handles
Richard Oudkerk added the comment:
The fact that that the FDs are duped shouldn't change anything to the
events reported: it works while the number of FDs is less than
FD_SETSIZE (epoll_wait() maxevents argument).
That assumes that epoll_wait() is supposed to return *all* ready fds
Richard Oudkerk added the comment:
Here is a version which uses epoll to service a number of pipes which is larger
than maxevents. (If NUM_WRITERS is too large then I get OSError: [Errno 24]
Too many open files.)
All pipes get serviced and the output is:
Working with 20 FDs, 5 maxevents
[5
Richard Oudkerk added the comment:
Yes, but the problem is that between two epoll_wait() calls, the
readiness of the FDs can have changed: and if that happens, you'll get
the same list over and over.
If an fd *was* ready but isn't anymore then why would you want to know about
it? Trying
Richard Oudkerk added the comment:
I don't like the idea of a specific I/O module for an OS. Is the public API
different?
It was partly to make integration with the existing tests easier: _io, _pyio
and winio are tested in parallel.
Can't you reuse the io module?
In what sense?
I don't
Richard Oudkerk added the comment:
Richard, in Tulip's WSAPoll code, it reads:
class WindowsPollPollster(PollPollster):
Pollster implementation using WSAPoll.
WSAPoll is only available on Windows Vista and later. Python
does not currently support WSAPoll
Richard Oudkerk added the comment:
try:
_MAXFD = os.sysconf(SC_OPEN_MAX)
-except:
+except ValueError:
_MAXFD = 256
os.sysconf() might raise OSError. I think ValueError is only raised if
_SC_OPEN_MAX was undefined when the module was compiled.
--
nosy: +sbt
Richard Oudkerk added the comment:
Why are you connecting to a multiprocessing listener with a raw socket? You
should be using multiprocessing.connection.Client to create a client connection.
Connection.send(obj) writes a 32 bit unsigned int (in network order) to the
socket representing
Richard Oudkerk added the comment:
Attached is a new patch which is implemented completely in C.
It adds a WinFileIO class to the io module, which has the same API
as FileIO except that:
* It has a handle attribute instead of a fileno() method.
* It has staticmethods openhandle
Changes by Richard Oudkerk shibt...@gmail.com:
Removed file: http://bugs.python.org/file28544/winfileio.c
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Changes by Richard Oudkerk shibt...@gmail.com:
Removed file: http://bugs.python.org/file28545/test_winfileio.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Changes by Richard Oudkerk shibt...@gmail.com:
Removed file: http://bugs.python.org/file28590/winfileio.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Richard Oudkerk added the comment:
Forgot to mention, the handles are non-inheritable.
You can use _winapi.DuplicateHandle() to create an inheritable duplicate handle
if you really need to.
--
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
If someone used regular sockets deliberately, they could crash
multiprocessing server code deliberately. Any chance of doing a real message
length check against the embedded message length check?
You can do
message = conn.recv_bytes(maxlength)
if you
Richard Oudkerk added the comment:
Thanks for the report.
It seems to only affect Windows, and only when using sockets rather than pipes.
Till this is fixed you could use
temp = bool(multiprocessing.connection.wait([cl], 1))
instead of
temp = cl.poll(1)
As I mentioned on the other
Richard Oudkerk added the comment:
The commits did not have the intended effect.
They just define a _poll() function (and only on Windows) and it is not
referenced anywhere else.
I will look in to fixing this -- on 2.7 and 3.2 this will need to be done in
the C code.
--
resolution
Richard Oudkerk added the comment:
This should be fixed now.
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
type: - behavior
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16955
Richard Oudkerk added the comment:
What do you mean? The intent was to use poll() instead of select()
anywhere available in order to avoid running out of fds.
The change didn't affect Windows because as of right now select() is
the only thing available.
The change *only* effects Windows
Richard Oudkerk added the comment:
It looks like the change to multiprocessing/connection.py committed does not
match the one uploaded as issue10527-3.patch
changeset 81174:e971a70984b8
1.1 --- a/Lib/multiprocessing/connection.py
1.2 +++ b/Lib/multiprocessing/connection.py
1.3
Richard Oudkerk added the comment:
Added some comments on Rietveld.
The .fileno() method is missing. Can this cause a problem when the file
is passed to stdlib functions? subprocess for example?
Thanks. An older version of the patch had a fileno() method which returned the
handle
Richard Oudkerk added the comment:
What does this proposal bring exactly?
Unless we are willing to completely replace fds with handles on Windows,
perhaps not too much. (At one point I had assumed that that was the plan for
py3k.)
Although not advertised, openhandle() does have
Richard Oudkerk added the comment:
If you want to communicate between processes of the same progam, you are best
off calling multiprocessing.Pipe() or multiprocessing.Queue() in the main
process. Queues or connections can then be inherited by the child processes.
Usually all communication
Richard Oudkerk added the comment:
For the reasons I wrote in the other issue, I don't think this an approach to
encourage.
There was no need to create a new issue: if you post to a closed issue then
people on the nosy list will still see your message.
So I will close this issue.
(Maybe
Richard Oudkerk added the comment:
New patch reflecting Amaury's comments.
--
Added file: http://bugs.python.org/file28737/winfileio.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Changes by Richard Oudkerk shibt...@gmail.com:
Removed file: http://bugs.python.org/file28707/winfileio.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12939
Richard Oudkerk added the comment:
I thought that access to the value field of Value instances was
protected by locks to avoid lost updates.
Loads and stores are both atomic. But += is made up of two operations, a
load followed by a store, and the lock is dropped between the two.
The same
Richard Oudkerk added the comment:
I see. Then this is a documentation bug. The examples in the
documentation use such non-thread-safe assignments (combined with the
statement These shared objects will be process and thread safe.).
Are you talking about this documentation:
If lock
Richard Oudkerk added the comment:
That compiles (after hacking the line endings). One Tulip test fails,
PollEventLooptests.testSockClientFail. But that's probably because the
PollSelector class hasn't been adjusted for Windows yet (need to dig this
out of the Pollster code
Richard Oudkerk added the comment:
On 21/01/2013 5:38pm, Guido van Rossum wrote:
This is a very good question to which I have no good answer.
If it weren't for this, we could probably do away with the distinction
between add_writer and add_connector, and a lot of code could be
simpler
Richard Oudkerk added the comment:
On 21/01/2013 7:00pm, Guido van Rossum wrote:
Regarding your IOCP changes, that sounds pretty exciting. Richard,
could you check those into the Tulip as a branch? (Maybe a new branch
named 'iocp'.)
OK. It may take me a while to rebase them
Richard Oudkerk added the comment:
I have created an iocp branch.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16507
___
___
Python-bugs-list
Richard Oudkerk added the comment:
It appears that Linux's spurious readiness notifications are a deliberate
deviation from the POSIX standard. (They are mentioned in the BUGS section of
the man page for select.)
Should I just apply the following patch to the default branch?
diff -r
Richard Oudkerk added the comment:
According to Alan Cox
It's a design decision and a huge performance win. It's one of the areas
where POSIX read in its strictest form cripples your performance.
See https://lkml.org/lkml/2011/6/18/103
(For write ready, you can obviously have
Richard Oudkerk added the comment:
For SOCK_STREAM, yes, not for SOCK_DGRAM
I thought SOCK_DGRAM messages just got truncated at the receiving end.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16507
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17018
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
On 27/01/2013 8:27pm, Terry J. Reedy wrote:
I agree we do not need to retain unpredictable 'dumb luck' -- in
future versions. But in the absence of a clear discrepancy
between doc and behavior (the definition of a bug) I believe
breaking such code
Richard Oudkerk added the comment:
On 27/01/2013 9:06pm, Serhiy Storchaka wrote:
Every bugfix breaks some code. As a compromise I propose set
m_obj-size = PY_SSIZE_T_MAX in case of overflow and emit a warning.
Trying to allocate PY_SSIZE_T_MAX bytes always seems to fail
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17097
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
Perhaps NEWS item needed for this change.
Done.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16743
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16743
Richard Oudkerk added the comment:
Richard, do you still want to push this forward? Otherwise I'd like to
finalize the patch (in the other sense ;-).
I started to worry a bit about daemon threads. I think they can still run
while atexit functions are being run.* So if a daemon thread
Richard Oudkerk added the comment:
On 14/02/2013 3:16pm, Antoine Pitrou wrote:
Mmmh... thread switching is already blocked at shutdown:
http://hg.python.org/cpython/file/0f827775f7b7/Python/ceval.c#l434
But in Py_Finalize(), call_py_exitfuncs() is called *before* _Py_Finalizing is
set
Changes by Richard Oudkerk shibt...@gmail.com:
--
status: pending - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16246
___
___
Python-bugs
Richard Oudkerk added the comment:
In any case, I think it's just something we'll have to live with. Daemon
threads are not a terrific idea in general.
I agree.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15528
Richard Oudkerk added the comment:
I did not realize there was a 'Extension Modules' section. I have been putting
changes to C extensions in the 'Library' section instead. It looks like most
people do the same as me.
--
nosy: +sbt
___
Python
Richard Oudkerk added the comment:
Was not it be yanked in 1fabff717ef4?
Looks like it was reintroduced by this merge changeset:
http://hg.python.org/cpython/rev/30fc620e240e
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Richard Oudkerk added the comment:
Good, except that you have to add a gc.collect() call for the
non-refcounted implementations.
Better to use test.support.gc_collect().
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http
Richard Oudkerk added the comment:
Banning md5 as a matter of policy may be perfectly sensible.
However, I think the way multiprocessing uses hmac authentication is *not*
affected by the collision attacks the advisory talks about. These depend on
the attacker being able to determine
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17261
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
A pool should only be used by the process that created it (unless you use a
managed pool).
If you are creating long lived processes then you could create a new pool on
demand. For example (untested)
pool_pid = (None, None)
def get_pool
Richard Oudkerk added the comment:
Richard, are you suggesting that we close this, or do you see an
actionable issue? (a plausible patch to the repository?)
I skimmed the documentation and could not see that this restriction has been
documented.
So I think a documentation patch would
Changes by Richard Oudkerk shibt...@gmail.com:
--
resolution: - fixed
stage: commit review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10527
Richard Oudkerk added the comment:
The new test seems to be reliably failing on Windows:
==
FAIL: test_issue17223 (__main__.UnicodeTest)
--
Traceback (most
Richard Oudkerk added the comment:
I think this change will potentially make the main module get imported twice
under different names when we transfer pickled data between processes. The
current code (which is rather a mess) goes out of its way to avoid that.
Basically the main process makes
Richard Oudkerk added the comment:
It looks like queues_contention.diff has the line
obj = pickle.dumps(obj)
in both _feed() and put(). Might that be why the third set of benchmarks was
slower than the second?
--
___
Python tracker rep
Richard Oudkerk added the comment:
On 04/03/2013 8:01pm, Charles-François Natali wrote:
It looks like queues_contention.diff has the line
obj = pickle.dumps(obj)
in both _feed() and put(). Might that be why the third set of benchmarks
was slower than the second?
_feed() is a Queue
Changes by Richard Oudkerk shibt...@gmail.com:
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17364
___
___
Python-bugs-list mailing
Richard Oudkerk added the comment:
The change in your patch is in a Windows-only section -- a few lines before the
chunk you can see _winapi.GetExitCodeProcess().
Since read() on Windows never fails with EINTR there is no need for
_eintr_retry_call().
If you are using Linux then there must
Richard Oudkerk added the comment:
BTW, on threads are only used on Windows. On Unix select() or poll() is used.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17367
Richard Oudkerk added the comment:
I will close the issue then.
If you track the problem down to a bug in Python then you can open a new one.
--
resolution: - invalid
stage: - committed/rejected
status: open - closed
___
Python tracker rep
Richard Oudkerk added the comment:
+1
To use Tools/builbot/*.bat doesn't the current directory have to be the main
directory of the repository? Then I see no point in the -C argument: just
set the correct directory automatically.
I think make.bat should also support creation of non-debug
Richard Oudkerk added the comment:
What does running 'kill-python before re-building python do? I have not
seen it mentioned in the in the devguide or pcbuild/readme.
It kills any currently running python(_d).exe processes. This is because
Windows does not allow program or library files
Richard Oudkerk added the comment:
LGTM (although the warning is actually harmless).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17395
Richard Oudkerk added the comment:
Which does give me a thought - perhaps lru_cache in 3.4 could accept a
key argument that is called as key(*args, **kwds) to derive the cache
key? (that would be a separate issue, of course)
Agreed. I suggested the same in an earlier post
Richard Oudkerk added the comment:
Why 1? This should be commented.
The manager process will always be included in active_children().
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17395
Richard Oudkerk added the comment:
Does this happen every time you run the tests? (I don't see these errors.)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17399
Richard Oudkerk added the comment:
Could you try the following program:
import socket
import multiprocessing
import multiprocessing.reduction
import multiprocessing.connection
def socketpair():
with socket.socket() as l:
l.bind(('localhost', 0))
l.listen(1)
s
Richard Oudkerk added the comment:
Now could you try the attached file? (It will not work on 2.7 because a
missing socket.fromfd().)
P.S. It looks like the error for 3.3 is associated with a file
f:\python\mypy\traceback.py which presumably clashes with the one in the
standard library
Richard Oudkerk added the comment:
Both 3.2 and 3.3 give essentially the same traceback as 3.2 did before,
both with installed python and yesterdays debug builds.
It looks like on your machine socket handles are not correctly inherited by
child processes -- I had assumed that they always
1 - 100 of 736 matches
Mail list logo
