Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-14 Thread Michael Baker 

On 14/04/2021 20:25, Jeff Peck wrote:


Just a complete shot in the dark. Have you verified that the 
environment variables you set are indeed getting picked up inside of 
your application? Try checking the output of os.GetEnv("HTTP_PROXY"), 
os.GetEnv("HTTPS_PROXY"), and os.GetEnv("NO_PROXY"). Make sure that if 
NO_PROXY is set that it isn't overriding your local proxy.



You could also try setting the lowercase versions of these env vars:

https://github.com/golang/net/blob/master/http/httpproxy/proxy.go

func FromEnvironment() *Config {
    return {
        HTTPProxy:  getEnvAny("HTTP_PROXY", "http_proxy"),
        HTTPSProxy: getEnvAny("HTTPS_PROXY", "https_proxy"),
        NoProxy:    getEnvAny("NO_PROXY", "no_proxy"),
        CGI:    os.Getenv("REQUEST_METHOD") != "",
    }
}

Thanks for the ideas Jeff. I already found out what the issue was though 
- see my other reply.






--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3b6c84a4-fed3-0bed-3c17-b595a3239066%40gmail.com.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-14 Thread Jeff Peck 
Just a complete shot in the dark. Have you verified that the environment 
variables you set are indeed getting picked up inside of your 
application? Try checking the output of os.GetEnv("HTTP_PROXY"), 
os.GetEnv("HTTPS_PROXY"), and os.GetEnv("NO_PROXY"). Make sure that if 
NO_PROXY is set that it isn't overriding your local proxy.



You could also try setting the lowercase versions of these env vars:

https://github.com/golang/net/blob/master/http/httpproxy/proxy.go

func FromEnvironment() *Config {
    return {
        HTTPProxy:  getEnvAny("HTTP_PROXY", "http_proxy"),
        HTTPSProxy: getEnvAny("HTTPS_PROXY", "https_proxy"),
        NoProxy:    getEnvAny("NO_PROXY", "no_proxy"),
        CGI:    os.Getenv("REQUEST_METHOD") != "",
    }
}


On 4/13/21 2:07 PM, Orson Cart wrote:

Please accept my ay apologies. I'd misunderstood the documentation.

I was looking for a reason why my go application's requests aren't 
being sent to the proxy that I've specified using HTTP_PROXY and 
HTTPS_PROXY. When I saw the comment in the documentation I thought I'd 
found an explanation even if I didn't understand the reason for it 
being that way.


So now I'm left with my original problem: I have HTTP_PROXY set to 
http://localhost: and HTTPS_PROXY set to https://localhost:. I 
also have a proxy (Fiddler) running on localhost: but requests 
from my application appear to go direct, bypassing the proxy.


I'm perplexed :(

On Tuesday, 13 April 2021 at 19:17:10 UTC+1 wagner riffel wrote:

On Tue Apr 13, 2021 at 2:14 PM -03, Orson Cart wrote:
> Can anyone explain the reasoning behind this? It rather
interferes with
> debugging traffic using a local proxy like Fiddler.
>

My guess here it's for preventing the remote from tricking the
proxy to
make request to internal services that couldn't be reached otherwise.

> I've seen suggestions to define an alternative hostname in
/etc/hosts
> but as far as I can tell this doesn't work either.
>

The code you linked doesn't seen to try that hard to block loopback
requests, I do think that an /etc/hosts entry to 127.0.0.1 should
bypass the proxy, if not I don't see any other way around other than
implementing the RoundTripper yourself, which shouldn't be hard for
such simple use.

--wagner

--
You received this message because you are subscribed to the Google 
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to golang-nuts+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ac6c1bae-2153-41b3-9037-110fc0eb12cfn%40googlegroups.com 
.


--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c889bc72-0a1d-72a1-83dc-ecafe5ef8458%40gmail.com.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-14 Thread Orson Cart 


On Wednesday, 14 April 2021 at 18:06:53 UTC+1 Adrian Ho wrote:

> The "Name Resolution" section in https://golang.org/pkg/net/ says:
>
> On Unix systems, the resolver has two options for resolving names. It can 
> use a pure Go resolver that sends DNS requests directly to the servers 
> listed in /etc/resolv.conf, or it can use a cgo-based resolver that calls C 
> library routines such as getaddrinfo and getnameinfo.
>
> By default the pure Go resolver is used, because a blocked DNS request 
> consumes only a goroutine, while a blocked C call consumes an operating 
> system thread.
>
> That might have some bearing on your problem. To see if it's the case, as 
> the page suggests:
>
> export GODEBUG=netdns=cgo   # force cgo resolver
>
>
> Best Regards,
> Adrian
>

Thanks Adrian. For the record, I'm working on Windows.

For the record I should also admit that I've been missing the blindingly 
obvious. At least it's blindingly obvious to me now.

I never occurred to me before that the client instance that I'm using has a 
non-default Transport. The way that the transport is initialised means that 
it has a nil Proxy - hence whatever I set in the environment didn't matter 
as it was being ignored.

This thread hasn't been my finest hour! 

Thanks to all for the help.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/16791ca0-702a-4474-9052-76ce67ba9a42n%40googlegroups.com.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-14 Thread Adrian Ho 

On 14/4/21 6:39 am, Orson Cart wrote:

So, to clarify:

1/ I have a proxy (Fiddler) running locally on port 
2/ my /etc/hosts file has an entry as follows: 127.0.0.1 fiddler
3/ My HTTP_PROXY and HTTPS_PROXY environment variables are set to 
http://fiddler:  and https://fiddler: 
 respectively


if I then issue a request via curl then that request is picked up by 
the proxy.
However if I make the same request from my go application, the request 
appears to bypass the proxy and is sent directly. I'm perplexed.


The "Name Resolution" section in https://golang.org/pkg/net/ says:

On Unix systems, the resolver has two options for resolving names. It 
can use a pure Go resolver that sends DNS requests directly to the 
servers listed in /etc/resolv.conf, or it can use a cgo-based resolver 
that calls C library routines such as getaddrinfo and getnameinfo.


By default the pure Go resolver is used, because a blocked DNS request 
consumes only a goroutine, while a blocked C call consumes an operating 
system thread.


That might have some bearing on your problem. To see if it's the case, 
as the page suggests:


export GODEBUG=netdns=cgo   # force cgo resolver


Best Regards,
Adrian

--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2709afa1-a037-8d3c-3fe0-f469588e3c8e%4003s.net.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-13 Thread Adrian Ho 

On 14/4/21 6:39 am, Orson Cart wrote:

So, to clarify:

1/ I have a proxy (Fiddler) running locally on port 
2/ my /etc/hosts file has an entry as follows: 127.0.0.1 fiddler
3/ My HTTP_PROXY and HTTPS_PROXY environment variables are set to 
http://fiddler:  and https://fiddler: 
 respectively


if I then issue a request via curl then that request is picked up by 
the proxy.
However if I make the same request from my go application, the request 
appears to bypass the proxy and is sent directly. I'm perplexed.


The "Name Resolution" section of https://golang.org/pkg/net/ says:

On Unix systems, the resolver has two options for resolving names. It 
can use a pure Go resolver that sends DNS requests directly to the 
servers listed in /etc/resolv.conf, or it can use a cgo-based resolver 
that calls C library routines such as getaddrinfo and getnameinfo.


By default the pure Go resolver is used, because a blocked DNS request 
consumes only a goroutine, while a blocked C call consumes an operating 
system thread.


That might have some bearing on your problem. To see if it's the case, 
as the page suggests:


export GODEBUG=netdns=cgo   # force cgo resolver


Best Regards,
Adrian

--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7c2a779f-90c7-a018-953f-221332e1fec3%4003s.net.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-13 Thread Orson Cart 
So, to clarify: 

1/ I have a proxy (Fiddler) running locally on port 
2/ my /etc/hosts file has an entry as follows: 127.0.0.1 fiddler
3/ My HTTP_PROXY and HTTPS_PROXY environment variables are set to 
http://fiddler:  and  https://fiddler: 
 respectively

if I then issue a request via curl then that request is picked up by the 
proxy.
However if I make the same request from my go application, the request 
appears to bypass the proxy and is sent directly. I'm perplexed.

Any advice would be appreciated.



On Tuesday, 13 April 2021 at 22:26:06 UTC+1 Orson Cart wrote:

> Thanks for the reply. 
>
> Yes I did add an alias to the /etc/hosts file and it makes no difference. 
> The requests still bypass the proxy.
>
> As you point out it's possible to achieve the goal in code but I'd really 
> like to do it without changing the code if at all possible.
>
> I'm also just really curious now as to why the go libraries aren't 
> honouring the environment variables.
>
> Thanks for the input.
>
> On Tuesday, 13 April 2021 at 21:36:48 UTC+1 wagner riffel wrote:
>
>> On Tue Apr 13, 2021 at 7:07 PM UTC, Orson Cart wrote:
>> > I'm perplexed :(
>> >
>>
>> Did you tried the /etc/hosts? I think that would do it, if not,
>> something like this may do the trick:
>>
>> if debug {
>> t := http.DefaultTransport.(*http.Transport)
>> t.Proxy = func(r *http.Request) (*url.URL, error) {
>> return url.Parse("http://localhost:;)
>> }
>> }
>>
>> note: i did not tested, just shooting.
>>
>> --wagner
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/708a6171-66bc-42bc-9284-0d46c7942d9dn%40googlegroups.com.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-13 Thread Orson Cart 
So, to clarify: With my HTTP_PROXY and HTTPS_PROXY environment variables 
set to http://localhost: and  https://localhost:  respectively, if 
I issue a request via curl then that request is picked up by the proxy.
However if I make the same request from my go application, the request 
appears to bypass the proxy and is sent directly.

Any advice would be appreciated.

On Tuesday, 13 April 2021 at 22:26:06 UTC+1 Orson Cart wrote:

> Thanks for the reply. 
>
> Yes I did add an alias to the /etc/hosts file and it makes no difference. 
> The requests still bypass the proxy.
>
> As you point out it's possible to achieve the goal in code but I'd really 
> like to do it without changing the code if at all possible.
>
> I'm also just really curious now as to why the go libraries aren't 
> honouring the environment variables.
>
> Thanks for the input.
>
> On Tuesday, 13 April 2021 at 21:36:48 UTC+1 wagner riffel wrote:
>
>> On Tue Apr 13, 2021 at 7:07 PM UTC, Orson Cart wrote:
>> > I'm perplexed :(
>> >
>>
>> Did you tried the /etc/hosts? I think that would do it, if not,
>> something like this may do the trick:
>>
>> if debug {
>> t := http.DefaultTransport.(*http.Transport)
>> t.Proxy = func(r *http.Request) (*url.URL, error) {
>> return url.Parse("http://localhost:;)
>> }
>> }
>>
>> note: i did not tested, just shooting.
>>
>> --wagner
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d6bd8344-59bb-4cd4-a9c5-3c1986d93724n%40googlegroups.com.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-13 Thread Orson Cart 
Thanks for the reply. 

Yes I did add an alias to the /etc/hosts file and it makes no difference. 
The requests still bypass the proxy.

As you point out it's possible to achieve the goal in code but I'd really 
like to do it without changing the code if at all possible.

I'm also just really curious now as to why the go libraries aren't 
honouring the environment variables.

Thanks for the input.

On Tuesday, 13 April 2021 at 21:36:48 UTC+1 wagner riffel wrote:

> On Tue Apr 13, 2021 at 7:07 PM UTC, Orson Cart wrote:
> > I'm perplexed :(
> >
>
> Did you tried the /etc/hosts? I think that would do it, if not,
> something like this may do the trick:
>
> if debug {
> t := http.DefaultTransport.(*http.Transport)
> t.Proxy = func(r *http.Request) (*url.URL, error) {
> return url.Parse("http://localhost:;)
> }
> }
>
> note: i did not tested, just shooting.
>
> --wagner
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/91dd7a03-4455-4fd6-854c-a588bd439925n%40googlegroups.com.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-13 Thread 'wagner riffel' via golang-nuts 
On Tue Apr 13, 2021 at 7:07 PM UTC, Orson Cart wrote:
> I'm perplexed :(
>

Did you tried the /etc/hosts? I think that would do it, if not,
something like this may do the trick:

if debug {
t := http.DefaultTransport.(*http.Transport)
t.Proxy = func(r *http.Request) (*url.URL, error) {
return url.Parse("http://localhost:;)
}
}

note: i did not tested, just shooting.

--wagner

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMVU7SLM2E5.16SWWYLIRF8HD%40pampas.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-13 Thread Orson Cart 
Please accept my ay apologies. I'd misunderstood the documentation.

I was looking for a reason why my go application's requests aren't being 
sent to the proxy that I've specified using HTTP_PROXY and HTTPS_PROXY. 
When I saw the comment in the documentation I thought I'd found an 
explanation even if I didn't understand the reason for it being that way.

So now I'm left with my original problem: I have HTTP_PROXY set to 
http://localhost: and HTTPS_PROXY set to https://localhost:. I also 
have a proxy (Fiddler) running on localhost: but requests from my 
application appear to go direct, bypassing the proxy.

I'm perplexed :(

On Tuesday, 13 April 2021 at 19:17:10 UTC+1 wagner riffel wrote:

> On Tue Apr 13, 2021 at 2:14 PM -03, Orson Cart wrote:
> > Can anyone explain the reasoning behind this? It rather interferes with
> > debugging traffic using a local proxy like Fiddler.
> >
>
> My guess here it's for preventing the remote from tricking the proxy to
> make request to internal services that couldn't be reached otherwise.
>
> > I've seen suggestions to define an alternative hostname in /etc/hosts
> > but as far as I can tell this doesn't work either.
> >
>
> The code you linked doesn't seen to try that hard to block loopback
> requests, I do think that an /etc/hosts entry to 127.0.0.1 should
> bypass the proxy, if not I don't see any other way around other than
> implementing the RoundTripper yourself, which shouldn't be hard for
> such simple use.
>
> --wagner
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ac6c1bae-2153-41b3-9037-110fc0eb12cfn%40googlegroups.com.

Re: [go-nuts] Why does Go ignore HTTP_PROXY and HTTPS_PROXY if the proxy address is localhost

2021-04-13 Thread 'wagner riffel' via golang-nuts 
On Tue Apr 13, 2021 at 2:14 PM -03, Orson Cart wrote:
> Can anyone explain the reasoning behind this? It rather interferes with
> debugging traffic using a local proxy like Fiddler.
>

My guess here it's for preventing the remote from tricking the proxy to
make request to internal services that couldn't be reached otherwise.

> I've seen suggestions to define an alternative hostname in /etc/hosts
> but as far as I can tell this doesn't work either.
>

The code you linked doesn't seen to try that hard to block loopback
requests, I do think that an /etc/hosts entry to 127.0.0.1 should
bypass the proxy, if not I don't see any other way around other than
implementing the RoundTripper yourself, which shouldn't be hard for
such simple use.

--wagner

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMSV7PV770P.3NLGXICF3GS3S%40pampas.