> On Apr 13, 2015, at 15:30, Chris Drane <[email protected]> wrote:
>
> I revisited the Stack Overflow post and it appears that I am able to receive
> HTTP responses properly now. Doing so required me to launch WireShark and
> copy a browser's actual headers. I also had to add a ContentDecoderAgent.
This is a good idea, but it's also a bit of an accident. Some sites may give
you length prefixes with a content decoder, but there will still be some that
provoke a PartialDownloadError no matter what you do. There are edge cases in
HTTP where you just cannot know if you have received everything the server
sent, and many sites still operate that way.
> Specifically it was the lack of an Accept-Encoding that was causing the
> problem. I added ["gzip, deflate, sdch"] and it seemed to do the trick.
Hrm. You have to set this header manually, even using a ContentDecoderAgent?
That sounds like a bug.
> I really do think that there should be an easier way to do this.
Absolutely. From the very beginning, Agent was not really supposed to be a
"high level" HTTP API, but those working on it sort of ran out of energy
halfway through. If you're writing applications today, you probably should use
treq instead - https://github.com/twisted/treq
<https://github.com/twisted/treq> - but longer-term the plan is to absorb treq
or something very much like it into Twisted itself.
Looking through the ticket tracker, though, I see that the plan ... does not
seem to be very well documented. I can't find the "high level" ticket
anywhere, and the closest thing I can find with just a few minutes of searching
the tracker is this:
https://twistedmatrix.com/trac/ticket/3987#comment:29
<https://twistedmatrix.com/trac/ticket/3987#comment:29>
which closes a ticket about a "high level interface" by talking about a
"mid-level API". We clearly need a higher level API within Twisted itself.
> I also don't understand why the Agent couldn't have properly interpreted the
> initial response.
It did properly interpret the initial response. You may or may not have
received the whole body, and that's exactly what PartialDownloadError means.
If responses which might be the whole response OR might be the server breaking
the connection on you are acceptable, handle that error and just treat the body
that you have received so far as complete. This is perfectly acceptable in
many cases.
> Thanks for everyone's time.
Thanks for using Twisted! Sorry that this experience was somewhat rocky. I
hope you'll stick around and help us improve it.
-glyph
_______________________________________________
Twisted-web mailing list
[email protected]
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
