[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: In patch v2 I integrated a version of the test into test_wsgiref. I swapped the sleep() calls for a threading.Event object. Because my patch is such a large change, it would be good to get other people’s opinions or reviews. -- Added file:

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: I did not write a test case for the test suite. Doing so would probably involve too many non-trivial things, so I thought it wouldn’t be worth it: 1. A background thread or process to run a client in 2. Signal handling to trigger a partial write 3. Some way to

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: wfile-partial.patch tries to avoid partial writes in every relevant wfile.write() call I could find. Usually I do this by building a BufferedWriter around wfile. I propose to apply my patch to 3.5 (if people think it is reasonable). For 3.6 I opened Issue

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: I suspect this bug does not affect Python 2. See my demo script wsgi-partial.py as evidence, which interrupts a 20 MB write with a signal. Jason: If you look at Python 2’s file.write() API , perhaps

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Jason Madden added the comment: I'm sorry, I'm still not following the argument that `write` is likely to return nothing. `RawIOBase` and `BufferedIOBase` both document that `write` must return the number of written bytes; if you don't return that, you break anything that assumes you do, as

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: My worry was that it is easy to make a write() method that does not return anything, but is still useful in most cases. Since BufferedIOBase.write() has to guarantee to write everything, it may not seem important to return a value. But we could explicitly

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Jason Madden added the comment: `self.stdin` and `self.stderr` are documented to be `wsgi.input` and `wsgi.errors`, which are both described as "file-like" objects, meaning that the `write` method should return bytes written. It seems like the same could reasonably be said to be true for

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Jason Madden added the comment: Is there an expected `self.stdout` implementation that doesn't return the number of bytes it writes? `sys.stdout` does, as does `io.RawIOBase`. It doesn't seem clear to me that practically speaking there's a compatibility problem with requiring that for

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: Okay now I see the conflict. The use of WSGIRequestHandler with wbufsize = 0 was the missing key. I see two possible solutions: 1. Change SimpleHandler._write() to allow self.stdout to be a RawIOBase writer, and loop over stdout.write() until all the data is

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Jason Madden added the comment: Django uses a `wsgiref.simple_server` to serve requests, which in turn uses `socketserver.StreamRequestHandler` to implement its `WSGIRequestHandler`. That base class explicitly turns off buffering for writes (`wbufsize = 0` is the class default which gets

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Jason Madden added the comment: gevent has another simple reproducer for this. I do believe it's not gevent's fault, the fault is in the standard library; SimpleHandler._write needs to loop until `sent += self.stdeout.write(data)` is `len(data)`. I have written up more on this at

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Changes by Martin Panter : -- stage: -> test needed versions: +Python 2.7 ___ Python tracker ___

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Paolo Veglia added the comment: I had the same issue as Jonathan with Python 3.5.1, big chunks of data get truncated (to ~85KB). The problem disappears when splitting original data into smaller chunks: def _write(self,data): chunk_size = 1024 for chunk in

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Marc Jofre added the comment: Hi all, I did more tests, identifying that the main issue I have resides in another script (multiprocessing), when using nuitka compiler. I will get back to you with more info when I solve the multiprocessing issue. The command prompt dump is: data: HTTP/1.0

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Marc Jofre added the comment: When executing, wsgiref produces the following error dump: D:\SixSensoCytometerWebBased\include\binD:\SixSensoCytometerWebBased\include\bi n\manage.exe runserver 8765 Validating models... 0 errors found May 26, 2015 - 16:20:56 Django version 1.5.8, using settings

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: Marc, if you think your connection aborted error is related, can you say what kind of object the “stdout” parameter is in the handler class? In other words, what class’s method is being called in the write() function at line 217? --

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: Actually, looking closer at Marc’s traceback, it appears to be the ordinary socket._fileobject class. So I suspect it is unrelated to this bug because the Python 2 socket code appears to call sendall() in a loop. --

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

New submission from Jonathan Kamens: The _write method of wsgiref.handlers.SimpleHandler reads as follows: def _write(self,data): self.stdout.write(data) The problem here is that calling write() on a socket is not actually guaranteed to write all of the data in the buffer. If the

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Changes by Ned Deily n...@acm.org: -- nosy: +pje versions: +Python 3.6 -Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24291 ___

[issue24291] wsgiref.handlers.SimpleHandler truncates large output blobs

Martin Panter added the comment: [UTF-8 error workaround] What kind of object is “stdout”? Plain Python socket objects don’t have a write() method. Perhaps “stdout” is meant to implement the BufferedIOBase.write() interface, which guarantees everything is written, even if it takes multiple