[issue22450] urllib doesn't put Accept: */* in the headers

Senthil Kumaran added the comment: @Martin, I weight in 'curl's behavior for de-facto things that differ slightly from standards. It's simply what folks have gotten used to, and sometimes expect. @Raymond, unit-tests will be a good addition too. --

[issue22450] urllib doesn't put Accept: */* in the headers

Changes by Raymond Hettinger : -- resolution: -> fixed status: open -> closed ___ Python tracker ___

[issue22450] urllib doesn't put Accept: */* in the headers

Roundup Robot added the comment: New changeset 00da8bfa2a60 by Raymond Hettinger in branch '3.5': Issue #22450: Use "Accept: */*" in the default headers for urllib.request https://hg.python.org/cpython/rev/00da8bfa2a60 -- ___ Python tracker

[issue22450] urllib doesn't put Accept: */* in the headers

Kenneth Reitz added the comment: I fully second Corey's comment. -- nosy: +kennethreitz ___ Python tracker ___

[issue22450] urllib doesn't put Accept: */* in the headers

Roundup Robot added the comment: New changeset e84105b48436 by Raymond Hettinger in branch '2.7': Issue #22450: Use "Accept: */*" in the default headers for urllib https://hg.python.org/cpython/rev/e84105b48436 -- nosy: +python-dev ___ Python tracker

[issue22450] urllib doesn't put Accept: */* in the headers

Martin Panter added the comment: I’m still not convinced. But my argument about the user specifying Accept if they care about the media type works both ways, so I am not that fussed if others want to make the change. Are there any examples of servers that behave worse than the

[issue22450] urllib doesn't put Accept: */* in the headers

Cory Benfield added the comment: So, leaping in on the Requests side of things for a moment, two notes. Firstly: copying curl is rarely a bad thing to do, especially for a behaviour curl has had for a long time. However, in this case the stronger argument is that just because the RFCs say

[issue22450] urllib doesn't put Accept: */* in the headers

Martin Panter added the comment: “Proxy servers such as NGinx and Varnish: . . . if the Accept header is omitted, the proxy cache can return any of the cached responses.” This is not really my area of expertise, but this behaviour is inconsistent with my understanding of how Accept and Vary

[issue22450] urllib doesn't put Accept: */* in the headers

Raymond Hettinger added the comment: Putting it another way: To an origin server, 'Accept: */*' means it can return anything it wants. To a proxy server, the absence of an accept header means in can return anything it has cached (possibly different from what the origin server would have

[issue22450] urllib doesn't put Accept: */* in the headers

Raymond Hettinger added the comment: Update: After more research, I learned that while 'Accept: */*' should not have an effect on the origin webserver, it can and does have an effect on proxy servers. Origin servers are allowed to vary the content-type of responses when given different

[issue22450] urllib doesn't put Accept: */* in the headers

Martin Panter added the comment: The Curl programmer replied basically saying there was no scientific reason, but since Curl was previously sending a custom Accept header, it was safer to leave a bare-bones Accept header in than completely remove it. Plus he thought it might be slighly more

[issue22450] urllib doesn't put Accept: */* in the headers

Raymond Hettinger added the comment: > What do you think, Raymond Before dismissing this, we should get a better understanding of why "Accept: */*" is so widely used in practice. Here's what we know so far: * The header made a difference to the Facebook Graph API. * Curl (a minimalist)

[issue22450] urllib doesn't put Accept: */* in the headers

Martin Panter added the comment: According to all the HTTP 1.1 RFCs, having */* at the end means you accept any other content type if none of the higher priority ones are available (otherwise you risk a 406 Not Acceptable error). So that explains why Firefox has */* tacked on. Requests

[issue22450] urllib doesn't put Accept: */* in the headers

Martin Panter added the comment: I propose rejecting this one, in favour of the caller adding their own “Accept: */*” (or more preferably, “Accept: application/json”) header. What do you think, Raymond or Senthil? -- ___ Python tracker

[issue22450] urllib doesn't put Accept: */* in the headers

Martin Panter added the comment: The RFC https://tools.ietf.org/html/rfc7231#page-39 says “A request without any Accept header field implies that the user agent will accept any media type in response”, which sounds the same as “Accept: */*”. I don’t understand why adding it should make a real

[issue22450] urllib doesn't put Accept: */* in the headers

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22450 ___

[issue22450] urllib doesn't put Accept: */* in the headers

New submission from Raymond Hettinger: The use of urllib for REST APIs is impaired in the absence of a Accept: */* header such as that added automatically by the requests package or by the CURL command-line tool. # Example that gets an incorrect result due to the missing header import

[issue22450] urllib doesn't put Accept: */* in the headers

Antoine Pitrou added the comment: Can you explain how the result is incorrect? f = urllib.request.urlopen('http://graph.facebook.com/raymondh') json.loads(f.read().decode()) {'link': 'https://www.facebook.com/raymondh', 'id': '562805507', 'last_name': 'Hettinger', 'gender': 'male',

[issue22450] urllib doesn't put Accept: */* in the headers

Senthil Kumaran added the comment: Patch looks good. Will need similar addition in urllib2 and inclusion of tests. -- nosy: +orsenthil versions: +Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22450

[issue22450] urllib doesn't put Accept: */* in the headers

Senthil Kumaran added the comment: Well, the result with loading using json will be same. but without sending Accept */*. The content-type returned is text/javascript; charset=UTF-8 and with sending of Accept */* the content-type is set to application/json; charset=UTF-8 (which is more

[issue22450] urllib doesn't put Accept: */* in the headers

Antoine Pitrou added the comment: The content-type returned is text/javascript; charset=UTF-8 and with sending of Accept */* the content-type is set to application/json; charset=UTF-8 (which is more desirable). Is that a bug in urllib, or in Facebook's HTTP implementation? Frankly, we