[issue3243] Support iterable bodies in httplib

tzickel added the comment: This patch was opened for 2.7 but never applied there ? https://github.com/python/cpython/pull/10226 This causes a bug with requests HTTP library (and others as well as httplib) when you want to send an iterable object as POST data (with a non-chunked way), it

[issue3243] Support iterable bodies in httplib

Change by tzickel : -- pull_requests: +9540 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue3243] Support iterable bodies in httplib

Georg Brandl ge...@python.org added the comment: Anything left to do here, Senthil? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___ ___

[issue3243] Support iterable bodies in httplib

Changes by Georg Brandl ge...@python.org: -- priority: deferred blocker - critical ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___ ___

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: Hi Georg, In the previous comment, I had written that 'let me see if we have to accommodate those very special case where data can be a zero length string just to accommodate the mistake it was present in the earlier version.' More, I

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: Yes, I think we should close it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___ ___

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: Thanks for the note. I shall some details to the NEWS entry before the release. Closing this report. -- priority: critical - low status: open - closed ___ Python tracker rep...@bugs.python.org

[issue3243] Support iterable bodies in httplib

Changes by Senthil Kumaran orsent...@gmail.com: -- priority: low - normal ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___ ___

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: A mistake with Content-Length in the previous commit resolved in revision 87469. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: This is committed in r87399. Documentation and NEWS is added. Thanks for the patch and review comments. -- resolution: - fixed stage: needs patch - committed/rejected status: open - closed ___

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: Hello, sorry for commenting on a closed issue... but I think the documentation change is incorrect. In urllib.request.rst, it says data is a string. However as seen in the changes to test_urllib2.py, data must be a bytes object rather than a

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: Also, the patch for request.py contains a debug statement, print(data) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___

[issue3243] Support iterable bodies in httplib

Georg Brandl ge...@python.org added the comment: Good catch, fixed in r87400. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___ ___

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: Also, I am not familiar with the backward-comparability requirements of py3k, but orsenthil's patch will break py3k code that relies on data being a string, shouldn't this be mentioned somewhere? Finally, I do not understand why my proposed

[issue3243] Support iterable bodies in httplib

Georg Brandl ge...@python.org added the comment: Hmm, indeed: Senthil, could data be a string in earlier versions? If yes, the code should be changed to still allow that. (But after beta2 please, it's already tagged.) -- status: closed - open ___

[issue3243] Support iterable bodies in httplib

Georg Brandl ge...@python.org added the comment: Raising priority so that this gets sorted out before final. -- priority: normal - deferred blocker ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: Xuanji, Thanks for the comments on 'data' being bytes. I had just cared to add the feature information. I think that data detail should have been updated too. I think for your other two questions, we discussed it msg123051 - socket in

[issue3243] Support iterable bodies in httplib

Éric Araujo mer...@netwok.org added the comment: What is __read__ supposed to be? I don't think is required. The point is that Python does not define a __read__ magic method. Only read exists, on file objects. -- ___ Python tracker

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: eric: sorry, that has been fixed in issue_3243_py3k_7.patch -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: Give a try to this minor variation of the patch with tests added and let me know your review comments. -- Added file: http://bugs.python.org/file19890/Issue3243-4.patch ___ Python tracker

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: Senthil: +try: + self.sock.sendall(data) Indentation problem here. +if isinstance(data,str): +content_length = len(data) I'm not sure I understand. What does sending an unicode string mean?

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: orsenthil: Hi, i don't quite understand why iter() needs to be called explicitly on data? As I understand it, if data is an iterable then you can use a for loop on it directly. -- ___ Python tracker

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: attaching new patch. this implements the memoryview solution suggested by pitrou. but it does contain this thing: if not request.has_header('Content-length'): if (not hasattr(data, '__read__') and isinstance(data,

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: attaching new patch. this implements the memoryview solution suggested by pitrou. but it does contain this thing: if not request.has_header('Content-length'): if (not hasattr(data, '__read__') and isinstance(data,

[issue3243] Support iterable bodies in httplib

Changes by Xuanji Li xua...@gmail.com: Removed file: http://bugs.python.org/file19892/issue_3243_py3k_5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: if not request.has_header('Content-length'): if (not hasattr(data, '__read__') and What is __read__ supposed to be? 2) Can call len but not buffer: assume len == #bytes Why do you need it at all? --

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: On Wed, Dec 01, 2010 at 10:06:25AM +, Antoine Pitrou wrote: +try: + self.sock.sendall(data) Indentation problem here. I could notice it now. Shall fix it. +if isinstance(data,str): +

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: On Wed, Dec 01, 2010 at 02:56:56PM +, Xuanji Li wrote: orsenthil: Hi, i don't quite understand why iter() needs to be called explicitly on data? As I understand it, if data is an iterable then you can use a for loop on it directly.

Re: [issue3243] Support iterable bodies in httplib

On Wed, Dec 01, 2010 at 05:08:26PM +, Antoine Pitrou wrote: Antoine Pitrou pit...@free.fr added the comment: if not request.has_header('Content-length'): if (not hasattr(data, '__read__') and What is __read__ supposed to be? I don't think is required. The previous 2.x version

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: On Thu, Dec 02, 2010 at 02:19:10AM +, Senthil Kumaran wrote: On Wed, Dec 01, 2010 at 10:06:25AM +, Antoine Pitrou wrote: +try: + self.sock.sendall(data) Indentation problem here. I could notice it now.

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: 2) Can call len but not buffer: assume len == #bytes Why do you need it at all? Hmm, I'm looking at the the tests in urllib2 that fail if we omit this... in test_urllib2 there are tests that do this: req = Request(http://example.com/;, ) and

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: Updated patch after correcting the mistake (bytes vs str) in the previous one. -- Added file: http://bugs.python.org/file19900/Issue3243-6.patch ___ Python tracker rep...@bugs.python.org

Re: [issue3243] Support iterable bodies in httplib

On Thu, Dec 02, 2010 at 03:08:55AM +, Xuanji Li wrote: req = Request(http://example.com/;, ) That should be: req = Request(http://example.com/;, b) I updated some of those in the latest updated patch. ___ Python-bugs-list mailing list

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: And my version too... -- Added file: http://bugs.python.org/file19901/issue_3243_py3k_6.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: On Thu, Dec 02, 2010 at 03:16:53AM +, Xuanji Li wrote: And my version too... +if hasattr(data, '__len__') and not len(data): +request.add_unredirected_header('Content-length', '0') This is very

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: On Thu, Dec 02, 2010 at 03:20:05AM +, Xuanji Li wrote: Actually I don't think you can go around changing test_urllib2.py, they are after all regression tests... and surely some users will send as data. Think about it this way. In

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: thought of a better way, we can check if data is true; this will cover and b. this is in issue_3243_py3k_7.patch -- Added file: http://bugs.python.org/file19902/issue_3243_py3k_7.patch ___ Python

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: Glaring at my mistakes. There was wrong examples in the test_urllib2 which I just corrected. There were.. Expect for the difference in (it = iter(data) - Which I am seeking Except for ... --

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: pitrou: actually that seems a bit suspect now... you need to handle 'data' differently depending on its type, and while you can determine the type by finding out when 'data' throws certain exceptions, it doesn't seem like what exceptions were

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: pitrou: actually that seems a bit suspect now... you need to handle 'data' differently depending on its type, Yes, but you can't know all appropriate types in advance, so it's better to try and catch the TypeError. I don't understand your

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: I don't fully understand Lib/urllib/request.py either, I just ported it and ran the unittests... it seems like what it does is that if you send an iterator through as 'data' you can't know the length in advance, and rather than let the len(data)

[issue3243] Support iterable bodies in httplib

Davide Rizzo sor...@gmail.com added the comment: len(data) will raise anyway. No, it won't, if the iterable happens to be a sequence. -- nosy: +davide.rizzo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: len(data) will raise anyway. No, it won't, if the iterable happens to be a sequence. Well, it seems the patch is confused between iterable and iterator. Only iterators have a __next__, but they usually don't have a __len__. The patch should

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: davide: yeah, hasattr(lol, '__next__') == False, even though strings are Iterable; so for strings and other such sequences the len(data) line will be executed. So technically we shouldn't say No Content-Length specified for iterable body but we

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: Xuanji, a wording which does convey the approximate meaning is fine. I think, the Exception error messages will help the people based on the Context. - Lets have the ValueError raised from the urllib/request.py. Changing it to

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: One way to check that it's bytes-compatible is to take a memoryview of it: memoryview(babc) memory at 0x1cf5120 memoryview(bytearray(babc)) memory at 0x1cf55a0 memoryview(array.array('b', babc)) memory at 0x1cf52a0 memoryview([babc])

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: - Lets have the ValueError raised from the urllib/request.py. Changing it to isinstance(data,collections.Iterable) as Antoine suggested is okay here too. Xuanji is right: it's not. We want bytes to be accepted, and it's an iterable.

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: Answering to myself, sorry. memoryview() does return the right answer of whether the object supports the buffer interface, *however* it doesn't mean the len() will be right. For example, take an array.array of ints: memoryview(array.array(I,

[issue3243] Support iterable bodies in httplib

Éric Araujo mer...@netwok.org added the comment: hasattr(lol, '__next__') == False, even though strings are Iterable FYI, magic methods are looked up on the class, not on the instance. That’s why ABCs are the right thing to use here.

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: So, reading all your comments, I gather that my proposed patch for client.py which is try: self.sock.sendall(data) except TypeError: if isinstance(data, collections.Iterable): for d in t: self.sock.sendall(d)

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: just to confirm: we WANT array.array(I, [1,2,3]) to have a content- length of 12, right? Yes, since it will emit 12 bytes in the body (you could actually have a test for it). -- ___ Python tracker

[issue3243] Support iterable bodies in httplib

Senthil Kumaran orsent...@gmail.com added the comment: xuanji, the issue you stumbled upon was just fixed by Raymond for the report Issue10565. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: What a timely coincidence. I'll try out the change soon. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: Changed according to Raymond's suggestion. I realized that there are some classes (str, bytes, array.array) that are iterable but should not be handled by the iteration logic provided by catlee. eg: if we iterate through b'a message' we'd get a

[issue3243] Support iterable bodies in httplib

Antoine Pitrou pit...@free.fr added the comment: Thanks for the patch. First, you don't need to support str, since sockets only accept binary strings (not unicode). Second, I think it's simpler and more generic to do something like: try: self.sock.sendall(data) except TypeError:

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: pitrou: that sounds good. I attached another patch. -- Added file: http://bugs.python.org/file19879/issue_3243_py3k_3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: Hi, attached is a ported version of the patch that applies against py3k. -- nosy: +xuanji Added file: http://bugs.python.org/file19852/issue_3243_py3k.diff ___ Python tracker rep...@bugs.python.org

[issue3243] Support iterable bodies in httplib

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Instead of hasattr(str,'next') consider using isinstance(str, collections.Iterable) Also consider changing the variable name from the now overly type specific, str to something like source to indicate the

[issue3243] Support iterable bodies in httplib

Xuanji Li xua...@gmail.com added the comment: Hi Raymond, I assume you're referring to catlee's patch? 'str' has been changed to 'data' in python 3.2 While porting the patch I ran into this issue, which is that isinstance(str, collections.Iterable) doesn't behave exactly like

[issue3243] Support iterable bodies in httplib

Changes by Éric Araujo mer...@netwok.org: -- keywords: +easy resolution: accepted - stage: patch review - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___

[issue3243] Support iterable bodies in httplib

Changes by Senthil Kumaran orsent...@gmail.com: -- assignee: jhylton - orsenthil ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___ ___

[issue3243] Support iterable bodies in httplib

Éric Araujo mer...@netwok.org added the comment: The patch needs to be ported to the py3k branch. -- nosy: -BreamoreBoy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___

[issue3243] Support iterable bodies in httplib

Mark Lawrence breamore...@yahoo.co.uk added the comment: Could someone with knowledge of httplib please move this forward, all the comments I see are positive. -- nosy: +BreamoreBoy ___ Python tracker rep...@bugs.python.org

[issue3243] Support iterable bodies in httplib

Changes by Éric Araujo mer...@netwok.org: -- keywords: +needs review nosy: +orsenthil stage: unit test needed - patch review versions: -Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243

[issue3243] Support iterable bodies in httplib

Changes by Éric Araujo mer...@netwok.org: -- nosy: +merwok ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___ ___ Python-bugs-list mailing

[issue3243] Support iterable bodies in httplib

Ryan Coyner rcoy...@gmail.com added the comment: This patch and its tests still work. Any particular reason why it hasn't been adopted yet? -- nosy: +rcoyner ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243

[issue3243] Support iterable bodies in httplib

Changes by Daniel Diniz aja...@gmail.com: -- priority: - normal stage: - test needed versions: +Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3243 ___

[issue3243] Support iterable bodies in httplib

Jeremy Hylton jer...@alum.mit.edu added the comment: Seems like a reasonable feature request. I'm going to apply a variant of the patch in 3.1 first. -- assignee: - jhylton nosy: +jhylton resolution: - accepted ___ Python tracker

[issue3243] Support iterable bodies in httplib

Chris AtLee [EMAIL PROTECTED] added the comment: The attached patch implements this for python 2.7. It also adds support for iterable bodies in urllib2, where it is more generally useful. urllib2 enforces the presence of a Content-Length header in the request if the body is an iterable,

[issue3243] Support iterable bodies in httplib

New submission from Chris AtLee [EMAIL PROTECTED]: httplib should support requests whose bodies are iterable objects. This would facilitate doing large file uploads via HTTP since you wouldn't have to load the entire file into memory to create the request string. Index: Lib/httplib.py