[issue25838] Lib/httplib.py: Resend http request on server close connection

[Mikhail Gulyaev]

Mikhail Gulyaev added the comment:

> Why can’t you do the file rewinding yourself, when you retry the request?
Actually I'm not retry to send a request. This behavior I met for httplib 
library while I'm send a single request, httplib send it to exist connection 
and then recreates connection on failure and send it again. Maybe this doing 
tcp socket
For GET POST and DELETE I meet the same behavior

> if someone wrote code that expects data to be sent from a non-zero file 
> position.
Well in this case I'd prefer to trunkate file before send.

--
Added file: http://bugs.python.org/file41285/httplib.pcapng

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Martin Panter]

Martin Panter added the comment:

Sorry but I have a few concerns about your patch:

Why does this have to be in the HTTPConnection.send() method? Why can’t you do 
the file rewinding yourself, when you retry the request?

This could break compatibility if someone wrote code that expects data to be 
sent from a non-zero file position.

I think this would be new feature (for the next version of Python), and 
couldn’t be accepted as a bug fix for 2.7.

See also Issue 9740 (persistent HTTP client connections), where I suggested a 
few things you mentioned, including polling for an unsolicited response or 
closed connection, and reconnecting before sending a request.

Also beware that a general HTTP client shouldn’t automatically retry idempotent 
requests if there is a chance that the original request was already received. 
PUT is idempotent so that is okay. POST is not, however.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Mikhail Gulyaev]

Changes by Mikhail Gulyaev :


Added file: http://bugs.python.org/file41286/Выделение_058.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Mikhail Gulyaev]

Changes by Mikhail Gulyaev :


Added file: http://bugs.python.org/file41288/Выделение_061.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Mikhail Gulyaev]

Changes by Mikhail Gulyaev :


Added file: http://bugs.python.org/file41287/Выделение_059.png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[R. David Murray]

R. David Murray added the comment:

This patch is definitely invalid.  There is no requirement that data be 
seekable.

Are you saying that there insufficient support in http(lib) for an application 
to handle this server timeout?  Can you provide a program that demonstrates the 
problem (even better would be a unit test that simulates the server timeout).  
I'm thinking it is likely that it is the responsibility of the layer above 
http(lib) to handle this, but the problem isn't clear enough to me to be sure 
(Martin probably has a better understanding of it).

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Martin Panter]

Martin Panter added the comment:

As far as I understand, httplib should not be automatically reconnecting and 
re-sending requests. I still suspect your script may be causing the retry, but 
you are welcome to prove me wrong.

Can you clarify “send PUT request to closed socket”: is the local OS socket 
closed (file descriptor is released)? Or is it that the remote end of the 
connection has been shut down? If it is the remote end, in Python 2 usually you 
would see a BadStatusLine or some kind of socket.error exception, and nothing 
would be retried.

I think we really need to know what your script is doing to be able to help. 
For instance, in the 059 screen shot, what API calls were made to cause data to 
be initially sent (presumably from local port 40736), and then the reconnection 
(local port 40757) with more data?

--
stage:  -> test needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[R. David Murray]

R. David Murray added the comment:

The issue is definitely in httplib2, then.  You should open an issue on their 
bug tracker.

--
resolution:  -> third party
stage: test needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Mikhail Gulyaev]

Mikhail Gulyaev added the comment:

Thanks for attention

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Mikhail Gulyaev]

Mikhail Gulyaev added the comment:

You right I found who resend requests!

My request is goes through httplb2(which use httplib and resends request on 
failure), but the issue is that if request body contains file and then that 
file is read out and on retry there is nothing to read since we already read it 
in httplib. 
So what solution could you suggest for me? Is it some patch for httplib2 or 
totally my own troubles

This issue is on border of interacting httplib and httplib2
 - httplib sends request and reads out a file
 - httplib2 resends a request but file is already readed out

Will it be honest if we rereads file in httplib2? and could we able assume that 
readable object(hasattr(data,'read') == True) has also tell and seek methods

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Martin Panter]

Martin Panter added the comment:

Okay that makes a lot more sense! I agree that this should either be fixed in 
httplib2 or in your own script. The problem parallels Issue 5038, where 
urlopen() is used (rather than httplib2), and the request is retried after 
getting an authorization failure (rather than after a disconnection).

One option, if you can use Python 3.2+, might be to use a custom iterable 
object as the body. Then you get a hook to rewind the file every time it is 
iterated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Mikhail Gulyaev]

New submission from Mikhail Gulyaev:

Hello guys!

Recently I recived some strange behavior for sending http requests using httplib

My python script uses httplib and interacts with a web server which have a 
keep-alive timeout 5 seconds. Script makes PUT requests and sends files to 
server. At first time it works ok. Then after 5 seconds server closes 
connection. And then I doing second PUT request, that fails in two stages:
 - HTTPConnection trys to send PUT request to closed socket.
 - HTTPConnection reconnects to server and sends request again but no file 
sended.
This behavior was checked using wireshark and debug output of httplib.

The best solution IMHO would be to checking socket state on each request, and 
reconnects if needed - but it seems this issue has no unique solution.

So I offer a patch which simply rewinds file before sending if needed. 
hasattr(data,'tell') could be used in checking condition
---
diff -r 002d8b981128 Lib/httplib.py
--- a/Lib/httplib.pyWed Dec 09 19:44:30 2015 +0200
+++ b/Lib/httplib.pyFri Dec 11 12:59:47 2015 +0600
@@ -865,6 +865,7 @@
 blocksize = 8192
 if hasattr(data,'read') and not isinstance(data, array):
 if self.debuglevel > 0: print "sendIng a read()able"
+if data.tell() > 0: data.seek(0) # rewind for retry send file
 datablock = data.read(blocksize)
 while datablock:
 self.sock.sendall(datablock)
---

--
messages: 256204
nosy: gmixo
priority: normal
severity: normal
status: open
title: Lib/httplib.py: Resend http request on server close connection
type: enhancement
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25838] Lib/httplib.py: Resend http request on server close connection

[Mikhail Gulyaev]

Changes by Mikhail Gulyaev :


--
keywords: +patch
Added file: http://bugs.python.org/file41284/httplib.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com