[issue3243] Support iterable bodies in httplib

[tzickel]

tzickel  added the comment:

This patch was opened for 2.7 but never applied there ?

https://github.com/python/cpython/pull/10226

This causes a bug with requests HTTP library (and others as well as httplib) 
when you want to send an iterable object as POST data (with a non-chunked way), 
it works in Python 3 but not 2, and this effects behaviour and performance...

--
nosy: +tzickel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[tzickel]

Change by tzickel :


--
pull_requests: +9540

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Anything left to do here, Senthil?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Georg Brandl]

Changes by Georg Brandl ge...@python.org:


--
priority: deferred blocker - critical

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Hi Georg,

In the previous comment, I had written that

'let me see if we have to accommodate those very special case
where data can be a zero length string just to accommodate the mistake
it was present in the earlier version.'

More, I think about it, the more it seems to me that accommodating that
special wrong case is not required.

The decision boils down to this.

1. In py3k, data for POST should be bytes.

2. But urllib.request had a 'bug/hole' that when a zero length string
was passed as a data, it did not raise an exception. There were cases
in test_urllib2 where zero length string was passed. I argue that it
was more of a mistake than, what we actually wanted to test it.

Because there is NO practical scenario where Zero length data as a
POST is useful.

3. Now, with the introduction of this feature requested in this issue,
this zero length string would raise an Exception and would demand that
even if it is zero length, please send it as bytes.

IMO, this is correct behavior and we need not accommodate the previous
one. 

So, I would recommend closing this bug as Fixed without further change.
At most, a NEWS item can be added to explain point 2.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

Yes, I think we should close it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Thanks for the note. I shall some details to the NEWS entry before the release. 
Closing this report.

--
priority: critical - low
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Changes by Senthil Kumaran orsent...@gmail.com:


--
priority: low - normal

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

A mistake with Content-Length in the previous commit resolved in revision 87469.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

This is committed in r87399. Documentation and NEWS is added. Thanks for the 
patch and review comments.

--
resolution:  - fixed
stage: needs patch - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

Hello, sorry for commenting on a closed issue... but I think the documentation 
change is incorrect. In urllib.request.rst, it says data is a string. However 
as seen in the changes to test_urllib2.py, data must be a bytes object rather 
than a string object. I think we should reopen this issue and change the 
documentation.

Thanks!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

Also, the patch for request.py contains a debug statement, print(data)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Good catch, fixed in r87400.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

Also, I am not familiar with the backward-comparability requirements of py3k, 
but orsenthil's patch will break py3k code that relies on data being a string, 
shouldn't this be mentioned somewhere?

Finally, I do not understand why my proposed change, which is to add

+if not data:
+request.add_unredirected_header('Content-length', '0')


so that code that relies on being able to pass a blank string as data (for 
instance, the code in test_urllib2.py) is still able to do so.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Hmm, indeed: Senthil, could data be a string in earlier versions?

If yes, the code should be changed to still allow that.  (But after beta2 
please, it's already tagged.)

--
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Raising priority so that this gets sorted out before final.

--
priority: normal - deferred blocker

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Xuanji, Thanks for the comments on 'data' being bytes. I had just cared to add 
the feature information. I think that data detail should have been updated too. 

I think for your other two questions, we discussed it msg123051 - socket in 
py3k handles only bytes, sending string was wrong and test_urllib2 had the 
mistake in sending zero length strings which weren't detected.

However, let me see if we have to accommodate those very special case where 
data can be a zero length string just to accommodate the mistake it was present 
in the earlier version.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

 What is __read__ supposed to be?
 I don't think is required.

The point is that Python does not define a __read__ magic method.  Only read 
exists, on file objects.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

eric: sorry, that has been fixed in issue_3243_py3k_7.patch

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Give a try to this minor variation of the patch with tests added and let me 
know your review comments.

--
Added file: http://bugs.python.org/file19890/Issue3243-4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Senthil:

+try:
+   self.sock.sendall(data)

Indentation problem here.

+if isinstance(data,str):
+content_length = len(data)

I'm not sure I understand. What does sending an unicode string mean?

+# Check iterable body support
+def iterable_body():
+yield one
+yield two
+yield three

Iterables of strings? this doesn't seem supported in the patch.

Also, it would be nice if the tests checked that the sent data is as expected.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

orsenthil: Hi, i don't quite understand why iter() needs to be called 
explicitly on data? As I understand it, if data is an iterable then you can use 
a for loop on it directly.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

attaching new patch. this implements the memoryview solution suggested by 
pitrou. but it does contain this thing:

if not request.has_header('Content-length'):
if (not hasattr(data, '__read__') and 
isinstance(data, collections.Iterable)):
print(data,is an iterable)
try:
m = memoryview(data)
print(m.itemsize * len(m))
request.add_unredirected_header(
'Content-length', '%d' % (len(m) * m.itemsize))
except TypeError:
try:
request.add_unredirected_header(
'Content-length', '%d' % len(data))
except TypeError:
raise ValueError(
No Content-Length specified for iterable body)

why is it so nested? because data can support 3 different interfaces:

1) Buffer interface, in that case use memoryview to count bytes
2) Can call len but not buffer: assume len == #bytes
3) Iterable but cannot call len or memoryview: raise ValueError

I hope there is a simpler way...

--
Added file: http://bugs.python.org/file19891/issue_3243_py3k_5.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

attaching new patch. this implements the memoryview solution suggested by 
pitrou. but it does contain this thing:

if not request.has_header('Content-length'):
if (not hasattr(data, '__read__') and 
isinstance(data, collections.Iterable)):
print(data,is an iterable)
try:
m = memoryview(data)
print(m.itemsize * len(m))
request.add_unredirected_header(
'Content-length', '%d' % (len(m) * m.itemsize))
except TypeError:
try:
request.add_unredirected_header(
'Content-length', '%d' % len(data))
except TypeError:
raise ValueError(
No Content-Length specified for iterable body)

why is it so nested? because data can support 3 different interfaces:

1) Buffer interface, in that case use memoryview to count bytes
2) Can call len but not buffer: assume len == #bytes
3) Iterable but cannot call len or memoryview: raise ValueError

I hope there is a simpler way...

--
Added file: http://bugs.python.org/file19892/issue_3243_py3k_5.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Changes by Xuanji Li xua...@gmail.com:


Removed file: http://bugs.python.org/file19892/issue_3243_py3k_5.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 if not request.has_header('Content-length'):
 if (not hasattr(data, '__read__') and 

What is __read__ supposed to be?

 2) Can call len but not buffer: assume len == #bytes

Why do you need it at all?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

On Wed, Dec 01, 2010 at 10:06:25AM +, Antoine Pitrou wrote:
 +try:
 +   self.sock.sendall(data)
 
 Indentation problem here.

I could notice it now. Shall fix it.

 
 +if isinstance(data,str):
 +content_length = len(data)
 
 I'm not sure I understand. What does sending an unicode string mean?

That's my mistake with understanding, I just realized (again) that
socket.send, socket.sendall does only bytes. And we don't encode the
unicode code string to send as bytes too.

 +def iterable_body():
 +yield one
 +yield two
 +yield three
 
 Iterables of strings? this doesn't seem supported in the patch.

It should be:

+yield bone
+yield btwo
+yield bthree

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

On Wed, Dec 01, 2010 at 02:56:56PM +, Xuanji Li wrote:
 orsenthil: Hi, i don't quite understand why iter() needs to be
 called explicitly on data? As I understand it, if data is an
 iterable then you can use a for loop on it directly.
 

The reasoning I followed was, data is an Iterable (a collection) and
you get an Iterator by passing via iter(). And you send the items by
looping over the iterator.

Honestly, I am not sure if iter is needed here too. I thought it was
not needed too, when you determine it is an Iterable and iterate over
it using the for loop. But I kept the iter() method just to create an
instance and send it.

Antoine, which would be the correct/ better?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: [issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

On Wed, Dec 01, 2010 at 05:08:26PM +, Antoine Pitrou wrote:
 Antoine Pitrou pit...@free.fr added the comment:
  if not request.has_header('Content-length'):
  if (not hasattr(data, '__read__') and 
 
 What is __read__ supposed to be?

I don't think is required. The previous 2.x version patch was doing
this just to ensure that it is not file object and then it is a
sequence. (I could not understand why)

Now, when you determine that the sequence can be bytes, bytearray or
array.array then testing for memory view is enough. File objects
without Content-Length would raise an Exception too.

  2) Can call len but not buffer: assume len == #bytes
 
 Why do you need it at all?
 
Not required. For the same reason as above.
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

On Thu, Dec 02, 2010 at 02:19:10AM +, Senthil Kumaran wrote:
 On Wed, Dec 01, 2010 at 10:06:25AM +, Antoine Pitrou wrote:
  +try:
  +   self.sock.sendall(data)
  
  Indentation problem here.
 
 I could notice it now. Shall fix it.

Sorry, there was not any, if you viewed the patch online, it looked
as if there was, but not really. (I was surprised as how it could be
and not be caught by tests in the first place).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

  2) Can call len but not buffer: assume len == #bytes

 Why do you need it at all?

Hmm, I'm looking at the the tests in urllib2 that fail if we omit this... in 
test_urllib2 there are tests that do this:

req = Request(http://example.com/;, )

and they expect Content-Length to be set to 0...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Updated patch after correcting the mistake (bytes vs str) in the previous one.

--
Added file: http://bugs.python.org/file19900/Issue3243-6.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: [issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

On Thu, Dec 02, 2010 at 03:08:55AM +, Xuanji Li wrote:
 
 req = Request(http://example.com/;, )

That should be:

 req = Request(http://example.com/;, b)

I updated some of those in the latest updated patch.

___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

And my version too...

--
Added file: http://bugs.python.org/file19901/issue_3243_py3k_6.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

On Thu, Dec 02, 2010 at 03:16:53AM +, Xuanji Li wrote:
 And my version too...
 

+if hasattr(data, '__len__') and not len(data):
+request.add_unredirected_header('Content-length', '0')

This is very special case. It should not be so. There was wrong
examples in the test_urllib2 which I just corrected.

Expect for the difference in (it = iter(data) - Which I am seeking
some advice too). Rest of the things in both patches are same.

Thank you.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

On Thu, Dec 02, 2010 at 03:20:05AM +, Xuanji Li wrote:
 Actually I don't think you can go around changing test_urllib2.py,
 they are after all regression tests... and surely some users will
 send  as data.
 

Think about it this way. In py3k, socket.send can handle only bytes,
not string. So sending  as data to the socket.send is wrong.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

thought of a better way, we can check if data is true; this will cover  and 
b. this is in issue_3243_py3k_7.patch

--
Added file: http://bugs.python.org/file19902/issue_3243_py3k_7.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Glaring at my mistakes.

 There was wrong examples in the test_urllib2 which I just corrected.

There were..

 Expect for the difference in (it = iter(data) - Which I am seeking

Except for ...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

pitrou: actually that seems a bit suspect now... you need to handle 'data' 
differently depending on its type, and while you can determine the type by 
finding out when 'data' throws certain exceptions, it doesn't seem like what 
exceptions were meant for.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 pitrou: actually that seems a bit suspect now... you need to handle
 'data' differently depending on its type,

Yes, but you can't know all appropriate types in advance, so it's better
to try and catch the TypeError.

I don't understand your changes in Lib/urllib/request.py. len(data) will
raise anyway.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

I don't fully understand Lib/urllib/request.py either, I just ported it and ran 
the unittests... it seems like what it does is that if you send an iterator 
through as 'data' you can't know the length in advance, and rather than let the 
len(data) raise an exception catlee thought it's better to raise an exception 
to tell the user exactly why the code failed (ie, because the user sent an 
iterator and there's no way to meaningfully find the Content-Length of that).

As for the catching exceptions vs using isinstance: I thought about it for a 
while, I think something like this feels right to me:

  try:
  self.sock.sendall(data)
  except TypeError:

  if isinstance(data, collections.Iterable):
  for d in t:
  self.sock.sendall(d)
  else:
  raise TypeError(data should be a bytes-like object or an iterable, 
got %r % type(it))


anyway, calling iter(data) is equivalent to calling data.__iter__(), so 
catching the exception is equivalent to hasattr(data, '__iter__'), which is 
roughly the same as isinstance(data, collections.Iterable). so we try the most 
straightforward method (sending everything) then if that fails, data is either 
an iterator or a wrong type.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Davide Rizzo]

Davide Rizzo sor...@gmail.com added the comment:

 len(data) will raise anyway.

No, it won't, if the iterable happens to be a sequence.

--
nosy: +davide.rizzo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

  len(data) will raise anyway.
 
 No, it won't, if the iterable happens to be a sequence.

Well, it seems the patch is confused between iterable and iterator. Only
iterators have a __next__, but they usually don't have a __len__.

The patch should really check for iterables, so it should use:

if isinstance(data, collections.Iterable)
raise ValueError#etc.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

davide: yeah, hasattr(lol, '__next__') == False, even though strings are 
Iterable; so for strings and other such sequences the len(data) line will be 
executed. So technically we shouldn't say No Content-Length specified for 
iterable body but we should say No Content-Length specified for iterable body 
that is not a sequence. 

Basically, this whole patch (both parts of it) will be much better off iif 
there is a clean way to say a is an iterable but a is not a sequence, because 
even though b'this is a message' is Iterable, we want to treat it differently 
compared to, say, a generator object; we do NOT want to use the Iterator 
features (iter, next) of it, we want to use the sequencey features (by sending 
the whole chunk of it, by calling len)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Xuanji, a wording which does convey the approximate meaning is fine. I think, 
the Exception error messages will help the people based on the Context.

- Lets have the ValueError raised from the urllib/request.py. Changing it to 
isinstance(data,collections.Iterable) as Antoine suggested is okay here too.
- Same change for http.client code checking for an Iterable as specified in 
msg122905

tests are fine. Docs and NEWS should be updated.

Georg: Is it okay, if we push this feature in before Dec 4th, beta1 release?

--
nosy: +georg.brandl

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

One way to check that it's bytes-compatible is to take a memoryview of it:

 memoryview(babc)
memory at 0x1cf5120
 memoryview(bytearray(babc))
memory at 0x1cf55a0
 memoryview(array.array('b', babc))
memory at 0x1cf52a0

 memoryview([babc])
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: cannot make memory view because object does not have the buffer 
interface
 memoryview(abc)
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: cannot make memory view because object does not have the buffer 
interface

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 - Lets have the ValueError raised from the urllib/request.py. Changing
 it to isinstance(data,collections.Iterable) as Antoine suggested is
 okay here too.

Xuanji is right: it's not. We want bytes to be accepted, and it's an
iterable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Answering to myself, sorry. memoryview() does return the right answer of 
whether the object supports the buffer interface, *however* it doesn't mean the 
len() will be right. For example, take an array.array of ints:

 memoryview(array.array(I, [1,2,3]))
memory at 0x1cf5720
 len(array.array(I, [1,2,3]))
3
 len(memoryview(array.array(I, [1,2,3])))
3
 len(bytes(array.array(I, [1,2,3])))
12

len() returns 3 but the number of bytes written out by sendall() will really be 
12...

*However*, the right len can be calculated using the memoryview:

 m = memoryview(array.array(I, [1,2,3]))
 len(m) * m.itemsize
12

(without actually converting to a bytes object, so all this is cheap even for 
very large buffers)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

 hasattr(lol, '__next__') == False, even though strings are Iterable

FYI, magic methods are looked up on the class, not on the instance.  That’s why 
ABCs are the right thing to use here.  
http://docs.python.org/dev/reference/datamodel#special-method-names

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

So, reading all your comments, I gather that my proposed patch for client.py 
which is

  try:
  self.sock.sendall(data)
  except TypeError:
  if isinstance(data, collections.Iterable):
  for d in t:
  self.sock.sendall(d)
  else:
  raise TypeError(data should be a bytes-like object or an iterable, 
got %r % type(it))

is ok, because it avoids using hasattr to test for Iterable and avoids 
isinstance() to check for specific types (str, bytes...) but instead uses 
exceptions (as pitrou suggested)?

if that is ok with everyone, I just need to work more on request.py to remove 
the hasattr; I'll probably use the memoryview solution suggested by pitrou.

just to confirm: we WANT array.array(I, [1,2,3]) to have a content-length of 
12, right?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 just to confirm: we WANT array.array(I, [1,2,3]) to have a content-
 length of 12, right?

Yes, since it will emit 12 bytes in the body (you could actually have a test 
for it).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

xuanji, the issue you stumbled upon was just fixed by Raymond for the report 
Issue10565.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

What a timely coincidence. I'll try out the change soon.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

Changed according to Raymond's suggestion. I realized that there are some 
classes (str, bytes, array.array) that are iterable but should not be handled 
by the iteration logic provided by catlee. eg: if we iterate through b'a 
message' we'd get a bunch of ints and sending them is wrong. patch attached.

--
Added file: http://bugs.python.org/file19873/issue_3243_py3k_2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Thanks for the patch.

First, you don't need to support str, since sockets only accept binary strings 
(not unicode).

Second, I think it's simpler and more generic to do something like:

  try:
  self.sock.sendall(data)
  except TypeError:
  try:
  it = iter(data)
  except TypeError:
  raise TypeError(data should be a bytes-like object or 
  an iterable, got %r % type(it))
  for d in t:
  self.sock.sendall(d)

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

pitrou: that sounds good. I attached another patch.

--
Added file: http://bugs.python.org/file19879/issue_3243_py3k_3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

Hi, attached is a ported version of the patch that applies against py3k.

--
nosy: +xuanji
Added file: http://bugs.python.org/file19852/issue_3243_py3k.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

Instead of 
hasattr(str,'next')

consider using
isinstance(str, collections.Iterable)

Also consider changing the variable name from the now overly type specific, 
str to something like source to indicate the significance of the data, not 
its type.

--
nosy: +rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Xuanji Li]

Xuanji Li xua...@gmail.com added the comment:

Hi Raymond, I assume you're referring to catlee's patch? 'str' has been changed 
to 'data' in python 3.2

While porting the patch I ran into this issue, which is that isinstance(str, 
collections.Iterable) doesn't behave exactly like hasattr(str,'next')


 hasattr(lol, '__next__')
False
 isinstance(lol, collections.Iterable)
True

so using the isinstance method would actually match against strings, which iirc 
would make the program fail. I can confirm later if needed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Éric Araujo]

Changes by Éric Araujo mer...@netwok.org:


--
keywords: +easy
resolution: accepted - 
stage: patch review - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Senthil Kumaran]

Changes by Senthil Kumaran orsent...@gmail.com:


--
assignee: jhylton - orsenthil

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

The patch needs to be ported to the py3k branch.

--
nosy:  -BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Mark Lawrence]

Mark Lawrence breamore...@yahoo.co.uk added the comment:

Could someone with knowledge of httplib please move this forward, all the 
comments I see are positive.

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Éric Araujo]

Changes by Éric Araujo mer...@netwok.org:


--
keywords: +needs review
nosy: +orsenthil
stage: unit test needed - patch review
versions:  -Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Éric Araujo]

Changes by Éric Araujo mer...@netwok.org:


--
nosy: +merwok

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Ryan Coyner]

Ryan Coyner rcoy...@gmail.com added the comment:

This patch and its tests still work. Any particular reason why it hasn't been 
adopted yet?

--
nosy: +rcoyner

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Daniel Diniz]

Changes by Daniel Diniz aja...@gmail.com:


--
priority:  - normal
stage:  - test needed
versions: +Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Jeremy Hylton]

Jeremy Hylton jer...@alum.mit.edu added the comment:

Seems like a reasonable feature request.  I'm going to apply a variant
of the patch in 3.1 first.

--
assignee:  - jhylton
nosy: +jhylton
resolution:  - accepted

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Chris AtLee]

Chris AtLee [EMAIL PROTECTED] added the comment:

The attached patch implements this for python 2.7.  It also adds support
for iterable bodies in urllib2, where it is more generally useful. 
urllib2 enforces the presence of a Content-Length header in the request
if the body is an iterable, whereas httplib does not.

The patch also includes updates to docs and tests (which all pass as of
r67584 on my macbook)

--
keywords: +patch
Added file: http://bugs.python.org/file12241/python-3243.patch

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3243] Support iterable bodies in httplib

[Chris AtLee]

New submission from Chris AtLee [EMAIL PROTECTED]:

httplib should support requests whose bodies are iterable objects.  This
would facilitate doing large file uploads via HTTP since you wouldn't
have to load the entire file into memory to create the request string.

Index: Lib/httplib.py
===
--- Lib/httplib.py  (revision 64600)
+++ Lib/httplib.py  (working copy)
@@ -688,7 +688,12 @@
self.__state = _CS_IDLE

def send(self, str):
-Send `str' to the server.
+Send `str` to the server.
+
+``str`` can be a string object, a file-like object that supports
+a .read() method, or an iterable object that supports a .next()
+method.
+
if self.sock is None:
if self.auto_open:
self.connect()
@@ -710,6 +715,10 @@
while data:
self.sock.sendall(data)
data=str.read(blocksize)
+elif hasattr(str,'next'):
+if self.debuglevel  0: print sendIng an iterable
+for data in str:
+self.sock.sendall(data)
else:
self.sock.sendall(str)
except socket.error, v:

--
components: Library (Lib)
messages: 69014
nosy: catlee
severity: normal
status: open
title: Support iterable bodies in httplib
type: feature request
versions: Python 2.7

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3243
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com