Re: Python-requests seems to 404 with Django/Tasty-pie?
I just discovered that the requests library honors HTTP_PROXY, but does not
honor NO_PROXY.
Based on what you say below, I'm betting this is your problem, or at least
a part of it.
One solution is for you to explicitly unset HTTP_PROXY when you don't want
it.
Another is for you to specify a proxies dict for requests.
Good luck!
Best
Alex
On Tuesday, October 2, 2012 2:31:52 PM UTC-7, Victor Hooi wrote:
>
> heya,
>
> Thanks for the tips - you're probably right, I might need to whip out
> wireshark or something and see what exactly is going on.
>
> However, one thing I did notice - I normally have the http_proxy
> environment variable set, as we use a HTTP proxy at work.
>
> However, if I unset the http_proxy variable, Python requests suddenly
> seems to start working again.
>
> I tried to set the no_proxy variable, and put in localhost and 127.0.0.1
> in there - however, Python requests doesn't seem to respect that?
>
> Cheers,
> Victor
>
> On Tuesday, 2 October 2012 22:48:26 UTC+10, Cal Leeming [Simplicity Media
> Ltd] wrote:
>>
>> Hi Victor,
>>
>> I've had my fair share of exposure with python requests - so thought I'd
>> chime in.
>>
>> On first glance, this looks to be an issue with specifying the port
>> number into python-requests, doing so seems to send the entire "
>> http://localhost:8000/api/v1/host/?name__regex==json; as the
>> request. However, further analysis shows that might not be the case.
>>
>> Looking at the python requests code;
>> https://github.com/kennethreitz/requests/blob/develop/requests/models.py
>>
>> >>> urlparse.urlparse("http://localhost:8080/test/url?with=params;)
>> ParseResult(scheme='http', netloc='localhost:8080', path='/test/url',
>> params='', query='with=params', fragment='')
>>
>> It then sends this directly into urllib3 using connection_from_url();
>> https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py
>>
>> This then calls the following;
>> scheme, host, port = get_host(url)
>> if scheme == 'https':
>> return HTTPSConnectionPool(host, port=port, **kw)
>> else:
>> return HTTPConnectionPool(host, port=port, **kw)
>>
>> get_host -> parse_url()
>> https://github.com/shazow/urllib3/blob/master/urllib3/util.py
>>
>> Tracing through urllib3 finally gets to parse_url();
>>
>> >>> urllib3.util.parse_url("http://localhost:8080/test/url?with=params;)
>> Url(scheme='http', auth=None, host='localhost', port=8080,
>> path='/test/url', query='with=params', fragment=None)
>>
>> So, lets look at path_url() instead;
>> https://github.com/kennethreitz/requests/blob/develop/requests/models.py
>>
>> >>> lol = requests.get("
>> http://localhost:8000/api/v1/host/?name__regex==json;)
>> >>> lol.request.path_url
>> '/api/v1/host/?name__regex==json'
>>
>> Performing a test connection shows;
>>
>> foxx@test01.internal [~] > nc -p8000 -l
>> GET /api/v1/host/?name__regex==json HTTP/1.1
>> Host: localhost:8000
>> Accept-Encoding: identity, deflate, compress, gzip
>> Accept: */*
>> User-Agent: python-requests/0.11.1
>>
>> So, from what I can tell, python requests is functioning normally.
>>
>> Personally, I'd say get wireshark running, or use the nc trick shown
>> above, perform 1 request using curl and 1 using python requests, then
>> compare the request headers.
>>
>> Can't throw much more time at this, but hope the above helps
>>
>> Cal
>>
>> On Tue, Oct 2, 2012 at 8:54 AM, Victor Hooi wrote:
>>
>>> Hi,
>>>
>>> I have a Django app that's serving up a RESTful API using tasty-pie.
>>>
>>> I'm using Django's development runserver to test.
>>>
>>> When I access it via a browser it works fine, and using Curl also works
>>> fine:
>>>
>>> curl
>>> "http://localhost:8000/api/v1/host/?name__regex=format=json
"
>>>
>>>
>>> On the console with runserver, I see:
>>>
>>> [02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=format=json
HTTP/1.1" 200 2845
>>>
>>>
>>> However, when I try to use the Python requests module (
>>> http://docs.python-requests.org/en/latest/), I get a 404:
>>>
>>> >>> r = requests.get('
http://localhost:8000/api/v1/host/?name__regex==json')
>>> r
>>>
>>>
>>> or:
>>>
>>> >>> r = requests.get('
http://localhost:8000/api/v1/host/?name__regex=format=json
')
>>> r
>>>
>>>
>>> or:
>>>
>>> >>> payload = { 'format': 'json'}
>>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>>> r
>>> r.url
u'http://localhost:8000/api/v1?format=json'
>>>
>>>
>>> Also, on the Django runserver console, I see:
>>>
>>> [02/Oct/2012 17:25:01] "GET
http://localhost:8000/api/v1/host/?name__regex==json HTTP/1.1"
404 161072
>>>
>>>
>>> For some reason, when I use requests, runserver prints out the whole
>>> request URL, including localhost - but when I use the browser, or curl, it
>>> only prints out the
Re: Python-requests seems to 404 with Django/Tasty-pie?
Ahh!
By sheer coincidence, it was actually me that integrated the latest
SOCKS/proxy support you see in urllib3 lol.
Basically, the current proxy handling is quite badly broken and required a
bunch of fixes.
I submitted a stable set of patches for integration here;
https://github.com/shazow/urllib3/pull/68
This was later merged and branched by the core repo maintainer on the
following branch;
https://github.com/shazow/urllib3/tree/socks-proxy
Sadly, the project we were working on that required proxy access was
finished, and I wasn't able to contribute any more time.
There are a few ways to hack this into working.. most of them involve
monkey-patching the hell out of urllib3 ;)
You could maybe try patching up 'path_url', with something really hacky
like str.replace("http://localhost:8080;, "") (don't shoot me!)
Let me know how you get on
Cal
On Tue, Oct 2, 2012 at 10:31 PM, Victor Hooi wrote:
> heya,
>
> Thanks for the tips - you're probably right, I might need to whip out
> wireshark or something and see what exactly is going on.
>
> However, one thing I did notice - I normally have the http_proxy
> environment variable set, as we use a HTTP proxy at work.
>
> However, if I unset the http_proxy variable, Python requests suddenly
> seems to start working again.
>
> I tried to set the no_proxy variable, and put in localhost and 127.0.0.1
> in there - however, Python requests doesn't seem to respect that?
>
> Cheers,
> Victor
>
>
> On Tuesday, 2 October 2012 22:48:26 UTC+10, Cal Leeming [Simplicity Media
> Ltd] wrote:
>
>> Hi Victor,
>>
>> I've had my fair share of exposure with python requests - so thought I'd
>> chime in.
>>
>> On first glance, this looks to be an issue with specifying the port
>> number into python-requests, doing so seems to send the entire "
>> http://localhost:8000/api/v1/**host/?name__regex==json
>> **" as the request. However, further analysis shows that might not be
>> the case.
>>
>> Looking at the python requests code;
>> https://github.com/**kennethreitz/requests/blob/**
>> develop/requests/models.py
>>
>> >>> urlparse.urlparse("http://**localhost:8080/test/url?with=**params
>> ")
>> ParseResult(scheme='http', netloc='localhost:8080', path='/test/url',
>> params='', query='with=params', fragment='')
>>
>> It then sends this directly into urllib3 using connection_from_url();
>> https://github.com/shazow/**urllib3/blob/master/urllib3/**
>> connectionpool.py
>>
>> This then calls the following;
>> scheme, host, port = get_host(url)
>> if scheme == 'https':
>> return HTTPSConnectionPool(host, port=port, **kw)
>> else:
>> return HTTPConnectionPool(host, port=port, **kw)
>>
>> get_host -> parse_url()
>> https://github.com/shazow/**urllib3/blob/master/urllib3/**util.py
>>
>> Tracing through urllib3 finally gets to parse_url();
>>
>> >>> urllib3.util.parse_url("http:/**/localhost:8080/test/url?with=**
>> params ")
>> Url(scheme='http', auth=None, host='localhost', port=8080,
>> path='/test/url', query='with=params', fragment=None)
>>
>> So, lets look at path_url() instead;
>> https://github.com/**kennethreitz/requests/blob/**
>> develop/requests/models.py
>>
>> >>> lol = requests.get("http://**localhost:8000/api/v1/host/?**
>> name__regex==json
>> ")
>> >>> lol.request.path_url
>> '/api/v1/host/?name__regex=&**format=json'
>>
>> Performing a test connection shows;
>>
>> foxx@test01.internal [~] > nc -p8000 -l
>> GET /api/v1/host/?name__regex=&**format=json HTTP/1.1
>> Host: localhost:8000
>> Accept-Encoding: identity, deflate, compress, gzip
>> Accept: */*
>> User-Agent: python-requests/0.11.1
>>
>> So, from what I can tell, python requests is functioning normally.
>>
>> Personally, I'd say get wireshark running, or use the nc trick shown
>> above, perform 1 request using curl and 1 using python requests, then
>> compare the request headers.
>>
>> Can't throw much more time at this, but hope the above helps
>>
>> Cal
>>
>> On Tue, Oct 2, 2012 at 8:54 AM, Victor Hooi wrote:
>>
>>> Hi,
>>>
>>> I have a Django app that's serving up a RESTful API using tasty-pie.
>>>
>>> I'm using Django's development runserver to test.
>>>
>>> When I access it via a browser it works fine, and using Curl also works
>>> fine:
>>>
>>> curl "http://localhost:8000/api/v1/**host/?name__regex=format=**
json "
>>>
>>>
>>> On the console with runserver, I see:
>>>
>>> [02/Oct/2012 17:24:20] "GET
Re: Python-requests seems to 404 with Django/Tasty-pie?
heya,
Thanks for the tips - you're probably right, I might need to whip out
wireshark or something and see what exactly is going on.
However, one thing I did notice - I normally have the http_proxy
environment variable set, as we use a HTTP proxy at work.
However, if I unset the http_proxy variable, Python requests suddenly seems
to start working again.
I tried to set the no_proxy variable, and put in localhost and 127.0.0.1 in
there - however, Python requests doesn't seem to respect that?
Cheers,
Victor
On Tuesday, 2 October 2012 22:48:26 UTC+10, Cal Leeming [Simplicity Media
Ltd] wrote:
>
> Hi Victor,
>
> I've had my fair share of exposure with python requests - so thought I'd
> chime in.
>
> On first glance, this looks to be an issue with specifying the port number
> into python-requests, doing so seems to send the entire "
> http://localhost:8000/api/v1/host/?name__regex==json; as the
> request. However, further analysis shows that might not be the case.
>
> Looking at the python requests code;
> https://github.com/kennethreitz/requests/blob/develop/requests/models.py
>
> >>> urlparse.urlparse("http://localhost:8080/test/url?with=params;)
> ParseResult(scheme='http', netloc='localhost:8080', path='/test/url',
> params='', query='with=params', fragment='')
>
> It then sends this directly into urllib3 using connection_from_url();
> https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py
>
> This then calls the following;
> scheme, host, port = get_host(url)
> if scheme == 'https':
> return HTTPSConnectionPool(host, port=port, **kw)
> else:
> return HTTPConnectionPool(host, port=port, **kw)
>
> get_host -> parse_url()
> https://github.com/shazow/urllib3/blob/master/urllib3/util.py
>
> Tracing through urllib3 finally gets to parse_url();
>
> >>> urllib3.util.parse_url("http://localhost:8080/test/url?with=params;)
> Url(scheme='http', auth=None, host='localhost', port=8080,
> path='/test/url', query='with=params', fragment=None)
>
> So, lets look at path_url() instead;
> https://github.com/kennethreitz/requests/blob/develop/requests/models.py
>
> >>> lol = requests.get("
> http://localhost:8000/api/v1/host/?name__regex==json;)
> >>> lol.request.path_url
> '/api/v1/host/?name__regex==json'
>
> Performing a test connection shows;
>
> foxx@test01.internal [~] > nc -p8000 -l
> GET /api/v1/host/?name__regex==json HTTP/1.1
> Host: localhost:8000
> Accept-Encoding: identity, deflate, compress, gzip
> Accept: */*
> User-Agent: python-requests/0.11.1
>
> So, from what I can tell, python requests is functioning normally.
>
> Personally, I'd say get wireshark running, or use the nc trick shown
> above, perform 1 request using curl and 1 using python requests, then
> compare the request headers.
>
> Can't throw much more time at this, but hope the above helps
>
> Cal
>
> On Tue, Oct 2, 2012 at 8:54 AM, Victor Hooi > wrote:
>
>> Hi,
>>
>> I have a Django app that's serving up a RESTful API using tasty-pie.
>>
>> I'm using Django's development runserver to test.
>>
>> When I access it via a browser it works fine, and using Curl also works
>> fine:
>>
>> curl
>> "http://localhost:8000/api/v1/host/?name__regex=format=json
>>> "
>>
>>
>> On the console with runserver, I see:
>>
>> [02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=format=json
>>> HTTP/1.1" 200 2845
>>
>>
>> However, when I try to use the Python requests module (
>> http://docs.python-requests.org/en/latest/), I get a 404:
>>
>> >>> r = requests.get('
>>> http://localhost:8000/api/v1/host/?name__regex==json')
>>> >>> r
>>>
>>
>>
>> or:
>>
>> >>> r = requests.get('
>>> http://localhost:8000/api/v1/host/?name__regex=format=json
>>> ')
>>> >>> r
>>>
>>
>>
>> or:
>>
>> >>> payload = { 'format': 'json'}
>>> >>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>>> >>> r
>>>
>>> >>> r.url
>>> u'http://localhost:8000/api/v1?format=json'
>>
>>
>> Also, on the Django runserver console, I see:
>>
>> [02/Oct/2012 17:25:01] "GET
>>> http://localhost:8000/api/v1/host/?name__regex==json HTTP/1.1"
>>> 404 161072
>>
>>
>> For some reason, when I use requests, runserver prints out the whole
>> request URL, including localhost - but when I use the browser, or curl, it
>> only prints out the part *after* the hostname:port
>>
>> I'm assuming this is something to do with the encoding, user-agent or
>> request type it's sending? Why does runserver print out different URLs for
>> requests versus browser/curl?
>>
>> Cheers,
>> Victor
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/django-users/-/ycLjP71ciAEJ.
>> To post to this group, send email to django...@googlegroups.com
>> .
>> To unsubscribe from this group,
Re: Python-requests seems to 404 with Django/Tasty-pie?
Hi Victor,
I've had my fair share of exposure with python requests - so thought I'd
chime in.
On first glance, this looks to be an issue with specifying the port number
into python-requests, doing so seems to send the entire "
http://localhost:8000/api/v1/host/?name__regex==json; as the
request. However, further analysis shows that might not be the case.
Looking at the python requests code;
https://github.com/kennethreitz/requests/blob/develop/requests/models.py
>>> urlparse.urlparse("http://localhost:8080/test/url?with=params;)
ParseResult(scheme='http', netloc='localhost:8080', path='/test/url',
params='', query='with=params', fragment='')
It then sends this directly into urllib3 using connection_from_url();
https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py
This then calls the following;
scheme, host, port = get_host(url)
if scheme == 'https':
return HTTPSConnectionPool(host, port=port, **kw)
else:
return HTTPConnectionPool(host, port=port, **kw)
get_host -> parse_url()
https://github.com/shazow/urllib3/blob/master/urllib3/util.py
Tracing through urllib3 finally gets to parse_url();
>>> urllib3.util.parse_url("http://localhost:8080/test/url?with=params;)
Url(scheme='http', auth=None, host='localhost', port=8080,
path='/test/url', query='with=params', fragment=None)
So, lets look at path_url() instead;
https://github.com/kennethreitz/requests/blob/develop/requests/models.py
>>> lol = requests.get("
http://localhost:8000/api/v1/host/?name__regex==json;)
>>> lol.request.path_url
'/api/v1/host/?name__regex==json'
Performing a test connection shows;
foxx@test01.internal [~] > nc -p8000 -l
GET /api/v1/host/?name__regex==json HTTP/1.1
Host: localhost:8000
Accept-Encoding: identity, deflate, compress, gzip
Accept: */*
User-Agent: python-requests/0.11.1
So, from what I can tell, python requests is functioning normally.
Personally, I'd say get wireshark running, or use the nc trick shown above,
perform 1 request using curl and 1 using python requests, then compare the
request headers.
Can't throw much more time at this, but hope the above helps
Cal
On Tue, Oct 2, 2012 at 8:54 AM, Victor Hooi wrote:
> Hi,
>
> I have a Django app that's serving up a RESTful API using tasty-pie.
>
> I'm using Django's development runserver to test.
>
> When I access it via a browser it works fine, and using Curl also works
> fine:
>
> curl
> "http://localhost:8000/api/v1/host/?name__regex=format=json
>> "
>
>
> On the console with runserver, I see:
>
> [02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=format=json
>> HTTP/1.1" 200 2845
>
>
> However, when I try to use the Python requests module (
> http://docs.python-requests.org/en/latest/), I get a 404:
>
> >>> r = requests.get('
>> http://localhost:8000/api/v1/host/?name__regex==json')
>> >>> r
>>
>
>
> or:
>
> >>> r = requests.get('
>> http://localhost:8000/api/v1/host/?name__regex=format=json
>> ')
>> >>> r
>>
>
>
> or:
>
> >>> payload = { 'format': 'json'}
>> >>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>> >>> r
>>
>> >>> r.url
>> u'http://localhost:8000/api/v1?format=json'
>
>
> Also, on the Django runserver console, I see:
>
> [02/Oct/2012 17:25:01] "GET
>> http://localhost:8000/api/v1/host/?name__regex==json HTTP/1.1"
>> 404 161072
>
>
> For some reason, when I use requests, runserver prints out the whole
> request URL, including localhost - but when I use the browser, or curl, it
> only prints out the part *after* the hostname:port
>
> I'm assuming this is something to do with the encoding, user-agent or
> request type it's sending? Why does runserver print out different URLs for
> requests versus browser/curl?
>
> Cheers,
> Victor
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-users/-/ycLjP71ciAEJ.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
Python-requests seems to 404 with Django/Tasty-pie?
Hi,
I have a Django app that's serving up a RESTful API using tasty-pie.
I'm using Django's development runserver to test.
When I access it via a browser it works fine, and using Curl also works
fine:
curl "http://localhost:8000/api/v1/host/?name__regex=format=json;
On the console with runserver, I see:
[02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=format=json
> HTTP/1.1" 200 2845
However, when I try to use the Python requests module
(http://docs.python-requests.org/en/latest/), I get a 404:
>>> r =
> requests.get('http://localhost:8000/api/v1/host/?name__regex==json')
> >>> r
>
or:
>>> r =
> requests.get('http://localhost:8000/api/v1/host/?name__regex=format=json')
> >>> r
>
or:
>>> payload = { 'format': 'json'}
> >>> r = requests.get('http://localhost:8000/api/v1', params=payload)
> >>> r
>
> >>> r.url
> u'http://localhost:8000/api/v1?format=json'
Also, on the Django runserver console, I see:
[02/Oct/2012 17:25:01] "GET
> http://localhost:8000/api/v1/host/?name__regex==json HTTP/1.1" 404
> 161072
For some reason, when I use requests, runserver prints out the whole
request URL, including localhost - but when I use the browser, or curl, it
only prints out the part *after* the hostname:port
I'm assuming this is something to do with the encoding, user-agent or
request type it's sending? Why does runserver print out different URLs for
requests versus browser/curl?
Cheers,
Victor
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/django-users/-/ycLjP71ciAEJ.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
