Re: resolvers - resolv.conf fallback

[Ben Draut]

Yep, will do.

On Tue, Apr 17, 2018 at 8:04 AM, Baptiste  wrote:

>
> On Sat, Apr 14, 2018 at 5:39 AM, Jonathan Matthews <
> cont...@jpluscplusm.com> wrote:
>
>> On 14 April 2018 at 05:13, Willy Tarreau  wrote:
>> > On Fri, Apr 13, 2018 at 03:48:19PM -0600, Ben Draut wrote:
>> >> How about 'parse-resolv-conf' for the current feature, and we reserve
>> >> 'use-system-resolvers' for the feature that Jonathan described?
>> >
>> > Perfect! "parse" is quite explicit at least!
>>
>> Works for me :-)
>>
>>
> Great, amazing!!!
> Ben, could you provide a patch using native code? (no third party
> libraries)
>
> Baptiste
>

Re: resolvers - resolv.conf fallback

[Baptiste]

On Sat, Apr 14, 2018 at 5:39 AM, Jonathan Matthews 
wrote:

> On 14 April 2018 at 05:13, Willy Tarreau  wrote:
> > On Fri, Apr 13, 2018 at 03:48:19PM -0600, Ben Draut wrote:
> >> How about 'parse-resolv-conf' for the current feature, and we reserve
> >> 'use-system-resolvers' for the feature that Jonathan described?
> >
> > Perfect! "parse" is quite explicit at least!
>
> Works for me :-)
>
>
Great, amazing!!!
Ben, could you provide a patch using native code? (no third party libraries)

Baptiste

Re: resolvers - resolv.conf fallback

[Jonathan Matthews]

On 14 April 2018 at 05:13, Willy Tarreau  wrote:
> On Fri, Apr 13, 2018 at 03:48:19PM -0600, Ben Draut wrote:
>> How about 'parse-resolv-conf' for the current feature, and we reserve
>> 'use-system-resolvers' for the feature that Jonathan described?
>
> Perfect! "parse" is quite explicit at least!

Works for me :-)

Re: resolvers - resolv.conf fallback

[Willy Tarreau]

On Fri, Apr 13, 2018 at 03:48:19PM -0600, Ben Draut wrote:
> How about 'parse-resolv-conf' for the current feature, and we reserve
> 'use-system-resolvers' for the feature that Jonathan described?

Perfect! "parse" is quite explicit at least!

Willy

Re: resolvers - resolv.conf fallback

[Ben Draut]

How about 'parse-resolv-conf' for the current feature, and we reserve
'use-system-resolvers' for the feature that Jonathan described?

On Fri, Apr 13, 2018 at 3:46 PM, Willy Tarreau  wrote:

> Hi Jonathan,
>
> On Fri, Apr 13, 2018 at 09:24:54PM +, Jonathan Matthews wrote:
> > On Fri, 13 Apr 2018 at 15:09, Willy Tarreau  wrote:
> >
> > > On Fri, Apr 13, 2018 at 08:01:13AM -0600, Ben Draut wrote:
> > > > How about this:
> > > >
> > > > * New directive: 'use_system_nameservers'
> > >
> > > OK, just use dashes ('-') instead of underscores as this is what we
> mostly
> > > use on other keywords, except a few historical mistakes.
> >
> >
> > I'm *definitely* not trying to bikeshed here, but from an Ops
> perspective a
> > reasonable implication of  "use_system_nameservers" would be for the
> > resolution process to track the currently configured contents of
> > resolv.conf over time.
> >
> > AIUI this will actually parse once, at proxy startup, which I suggest
> > should be made more obvious in the naming.
>
> That's a very good point. Maybe "same-as-resolv-conf" or something
> similar would make it more obvious ?
>
> > If I'm wrong, or splitting hairs, please ignore!
>
> Better split hair before features get released and we confuse them all
> the time! There are a few names I'm not proud of you know ;-)
>
> Willy
>

Re: resolvers - resolv.conf fallback

[Willy Tarreau]

Hi Jonathan,

On Fri, Apr 13, 2018 at 09:24:54PM +, Jonathan Matthews wrote:
> On Fri, 13 Apr 2018 at 15:09, Willy Tarreau  wrote:
> 
> > On Fri, Apr 13, 2018 at 08:01:13AM -0600, Ben Draut wrote:
> > > How about this:
> > >
> > > * New directive: 'use_system_nameservers'
> >
> > OK, just use dashes ('-') instead of underscores as this is what we mostly
> > use on other keywords, except a few historical mistakes.
> 
> 
> I'm *definitely* not trying to bikeshed here, but from an Ops perspective a
> reasonable implication of  "use_system_nameservers" would be for the
> resolution process to track the currently configured contents of
> resolv.conf over time.
> 
> AIUI this will actually parse once, at proxy startup, which I suggest
> should be made more obvious in the naming.

That's a very good point. Maybe "same-as-resolv-conf" or something
similar would make it more obvious ?

> If I'm wrong, or splitting hairs, please ignore!

Better split hair before features get released and we confuse them all
the time! There are a few names I'm not proud of you know ;-)

Willy

Re: resolvers - resolv.conf fallback

[Jonathan Matthews]

On Fri, 13 Apr 2018 at 15:09, Willy Tarreau  wrote:

> On Fri, Apr 13, 2018 at 08:01:13AM -0600, Ben Draut wrote:
> > How about this:
> >
> > * New directive: 'use_system_nameservers'
>
> OK, just use dashes ('-') instead of underscores as this is what we mostly
> use on other keywords, except a few historical mistakes.


I'm *definitely* not trying to bikeshed here, but from an Ops perspective a
reasonable implication of  "use_system_nameservers" would be for the
resolution process to track the currently configured contents of
resolv.conf over time.

AIUI this will actually parse once, at proxy startup, which I suggest
should be made more obvious in the naming.

If I'm wrong, or splitting hairs, please ignore!

J

> --
Jonathan Matthews
London, UK
http://www.jpluscplusm.com/contact.html

Re: resolvers - resolv.conf fallback

[Willy Tarreau]

On Fri, Apr 13, 2018 at 08:01:13AM -0600, Ben Draut wrote:
> How about this:
> 
> * New directive: 'use_system_nameservers'

OK, just use dashes ('-') instead of underscores as this is what we mostly
use on other keywords, except a few historical mistakes.

> * Parse /etc/resolv.conf for nameservers when new directive is seen when
> parsing resolvers.

OK.

> * Make check_config_validity issue a warning for each resolvers section
> that has an empty nameservers list.

OK.

> Thoughts?

All of this sounds perfect.

Thanks,
Willy

Re: resolvers - resolv.conf fallback

[Ben Draut]

How about this:

* New directive: 'use_system_nameservers'
* Parse /etc/resolv.conf for nameservers when new directive is seen when
parsing resolvers.
* Make check_config_validity issue a warning for each resolvers section
that has an empty nameservers list.

Thoughts?

On Tue, Apr 10, 2018 at 3:12 PM, Ben Draut  wrote:

> I agree.
>
> On Mon, Apr 9, 2018 at 1:35 AM, Baptiste  wrote:
>
>>
>>
>> On Fri, Apr 6, 2018 at 4:54 PM, Willy Tarreau  wrote:
>>
>>> On Fri, Apr 06, 2018 at 04:50:54PM +0200, Lukas Tribus wrote:
>>> > > Well, sometimes when you're debugging a configuration, it's nice to
>>> be
>>> > > able to disable some elements. Same for those manipulating/building
>>> > > configs by assembling elements and iteratively pass them through
>>> > > "haproxy -c". That's exactly the reason why we relaxed a few checks
>>> in
>>> > > the past, like accepting a frontend with no bind line or accepting a
>>> > > backend with a "cookie" directive with no cookie on server lines. In
>>> > > fact we could simply emit a warning when a resolvers section has no
>>> > > resolver nor resolv.conf enabled, but at least accept to start.
>>> >
>>> > Understood; however in this specific case I would argue one would
>>> > remove the "resolver" directive from the server-line(s), instead of
>>> > dropping the nameservers from the global nameserver declaration.
>>>
>>> No, because in order to do this, you also have to remove all references
>>> on all "server" lines, which is quite a pain, and error-prone when you
>>> want to reactivate them.
>>>
>>> > Maybe a config warning would be a compromise for this case?
>>>
>>> Yes, that's what I mentionned above, I'm all in favor of this given that
>>> we can't objectively find a valid use case for an empty resolvers section
>>> in production.
>>>
>>> Cheers,
>>> Willy
>>>
>>
>>
>> Ok, so just to summarize:
>> - we should enable parsing of resolv.conf with a configuration statement
>> in the resolvers section
>> - only nameserver directives from resolv.conf will be parsed for now
>> - parsing of resolv.conf can be used in conjunction with nameserver
>> directives in the resolvers section
>> - HAProxy should emit a warning message when parsing a configuration
>> which has no resolv.conf neither nameserver directives enabled
>>
>> Is that correct?
>>
>> Baptiste
>>
>
>

Re: resolvers - resolv.conf fallback

[Ben Draut]

I agree.

On Mon, Apr 9, 2018 at 1:35 AM, Baptiste  wrote:

>
>
> On Fri, Apr 6, 2018 at 4:54 PM, Willy Tarreau  wrote:
>
>> On Fri, Apr 06, 2018 at 04:50:54PM +0200, Lukas Tribus wrote:
>> > > Well, sometimes when you're debugging a configuration, it's nice to be
>> > > able to disable some elements. Same for those manipulating/building
>> > > configs by assembling elements and iteratively pass them through
>> > > "haproxy -c". That's exactly the reason why we relaxed a few checks in
>> > > the past, like accepting a frontend with no bind line or accepting a
>> > > backend with a "cookie" directive with no cookie on server lines. In
>> > > fact we could simply emit a warning when a resolvers section has no
>> > > resolver nor resolv.conf enabled, but at least accept to start.
>> >
>> > Understood; however in this specific case I would argue one would
>> > remove the "resolver" directive from the server-line(s), instead of
>> > dropping the nameservers from the global nameserver declaration.
>>
>> No, because in order to do this, you also have to remove all references
>> on all "server" lines, which is quite a pain, and error-prone when you
>> want to reactivate them.
>>
>> > Maybe a config warning would be a compromise for this case?
>>
>> Yes, that's what I mentionned above, I'm all in favor of this given that
>> we can't objectively find a valid use case for an empty resolvers section
>> in production.
>>
>> Cheers,
>> Willy
>>
>
>
> Ok, so just to summarize:
> - we should enable parsing of resolv.conf with a configuration statement
> in the resolvers section
> - only nameserver directives from resolv.conf will be parsed for now
> - parsing of resolv.conf can be used in conjunction with nameserver
> directives in the resolvers section
> - HAProxy should emit a warning message when parsing a configuration which
> has no resolv.conf neither nameserver directives enabled
>
> Is that correct?
>
> Baptiste
>

Re: resolvers - resolv.conf fallback

[Baptiste]

On Fri, Apr 6, 2018 at 4:54 PM, Willy Tarreau  wrote:

> On Fri, Apr 06, 2018 at 04:50:54PM +0200, Lukas Tribus wrote:
> > > Well, sometimes when you're debugging a configuration, it's nice to be
> > > able to disable some elements. Same for those manipulating/building
> > > configs by assembling elements and iteratively pass them through
> > > "haproxy -c". That's exactly the reason why we relaxed a few checks in
> > > the past, like accepting a frontend with no bind line or accepting a
> > > backend with a "cookie" directive with no cookie on server lines. In
> > > fact we could simply emit a warning when a resolvers section has no
> > > resolver nor resolv.conf enabled, but at least accept to start.
> >
> > Understood; however in this specific case I would argue one would
> > remove the "resolver" directive from the server-line(s), instead of
> > dropping the nameservers from the global nameserver declaration.
>
> No, because in order to do this, you also have to remove all references
> on all "server" lines, which is quite a pain, and error-prone when you
> want to reactivate them.
>
> > Maybe a config warning would be a compromise for this case?
>
> Yes, that's what I mentionned above, I'm all in favor of this given that
> we can't objectively find a valid use case for an empty resolvers section
> in production.
>
> Cheers,
> Willy
>


Ok, so just to summarize:
- we should enable parsing of resolv.conf with a configuration statement in
the resolvers section
- only nameserver directives from resolv.conf will be parsed for now
- parsing of resolv.conf can be used in conjunction with nameserver
directives in the resolvers section
- HAProxy should emit a warning message when parsing a configuration which
has no resolv.conf neither nameserver directives enabled

Is that correct?

Baptiste

Re: resolvers - resolv.conf fallback

[Willy Tarreau]

On Fri, Apr 06, 2018 at 04:50:54PM +0200, Lukas Tribus wrote:
> > Well, sometimes when you're debugging a configuration, it's nice to be
> > able to disable some elements. Same for those manipulating/building
> > configs by assembling elements and iteratively pass them through
> > "haproxy -c". That's exactly the reason why we relaxed a few checks in
> > the past, like accepting a frontend with no bind line or accepting a
> > backend with a "cookie" directive with no cookie on server lines. In
> > fact we could simply emit a warning when a resolvers section has no
> > resolver nor resolv.conf enabled, but at least accept to start.
> 
> Understood; however in this specific case I would argue one would
> remove the "resolver" directive from the server-line(s), instead of
> dropping the nameservers from the global nameserver declaration.

No, because in order to do this, you also have to remove all references
on all "server" lines, which is quite a pain, and error-prone when you
want to reactivate them.

> Maybe a config warning would be a compromise for this case?

Yes, that's what I mentionned above, I'm all in favor of this given that
we can't objectively find a valid use case for an empty resolvers section
in production.

Cheers,
Willy

Re: resolvers - resolv.conf fallback

[Lukas Tribus]

Hello Willy,


On 6 April 2018 at 14:14, Willy Tarreau  wrote:
>> The confusion often arises because haproxy accepts a resolver
>> configuration where no resolvers are configured. Maybe we should
>> reject the configuration when a resolver is referred to in the servers
>> lines, but no actual resolvers are configured (AND resolv.conf parsing
>> is not enabled in future).
>
> Well, sometimes when you're debugging a configuration, it's nice to be
> able to disable some elements. Same for those manipulating/building
> configs by assembling elements and iteratively pass them through
> "haproxy -c". That's exactly the reason why we relaxed a few checks in
> the past, like accepting a frontend with no bind line or accepting a
> backend with a "cookie" directive with no cookie on server lines. In
> fact we could simply emit a warning when a resolvers section has no
> resolver nor resolv.conf enabled, but at least accept to start.

Understood; however in this specific case I would argue one would
remove the "resolver" directive from the server-line(s), instead of
dropping the nameservers from the global nameserver declaration.

Maybe a config warning would be a compromise for this case?



Regards,

Lukas

Re: resolvers - resolv.conf fallback

[Willy Tarreau]

Hi Lukas,

On Fri, Apr 06, 2018 at 12:33:14PM +0200, Lukas Tribus wrote:
> On 6 April 2018 at 11:14, Willy Tarreau  wrote:
> >> I don't think we need a new config know.
> >
> > Just thinking, is the goal *not to have to* configure "resolve" on
> > server lines in this case, or to avoid having to pre-configure the
> > resolvers themselves when they're the same as the system's ?
> 
> The latter is the goal.

OK thanks for confirming.

> > If the former, that would mean always enabling DNS resolving at runtime
> > which doesn't sound like a good idea at all to me. If the latter, then
> > why not have a special directive in the resolvers section to indicate
> > that it should use resolv.conf instead ? That could avoid some surprizes
> > when you simply comment your all your resolvers and that suddenly the
> > behaviour changes. I'd even say that this directive could serve to
> > populate the resolvers section from resolv.conf (thus possibly several
> > resolvers) which will ensure the exclusivity between the two mechanisms.
> 
> Yes, that's a good point. In fact, I don't see why the fallback has to
> be implicit.
> 
> The confusion often arises because haproxy accepts a resolver
> configuration where no resolvers are configured. Maybe we should
> reject the configuration when a resolver is referred to in the servers
> lines, but no actual resolvers are configured (AND resolv.conf parsing
> is not enabled in future).

Well, sometimes when you're debugging a configuration, it's nice to be
able to disable some elements. Same for those manipulating/building
configs by assembling elements and iteratively pass them through
"haproxy -c". That's exactly the reason why we relaxed a few checks in
the past, like accepting a frontend with no bind line or accepting a
backend with a "cookie" directive with no cookie on server lines. In
fact we could simply emit a warning when a resolvers section has no
resolver nor resolv.conf enabled, but at least accept to start.

Just my two cents,

Cheers,
Willy

Re: resolvers - resolv.conf fallback

[Lukas Tribus]

Hi Willy,



On 6 April 2018 at 11:14, Willy Tarreau  wrote:
>> I don't think we need a new config know.
>
> Just thinking, is the goal *not to have to* configure "resolve" on
> server lines in this case, or to avoid having to pre-configure the
> resolvers themselves when they're the same as the system's ?

The latter is the goal.



> If the former, that would mean always enabling DNS resolving at runtime
> which doesn't sound like a good idea at all to me. If the latter, then
> why not have a special directive in the resolvers section to indicate
> that it should use resolv.conf instead ? That could avoid some surprizes
> when you simply comment your all your resolvers and that suddenly the
> behaviour changes. I'd even say that this directive could serve to
> populate the resolvers section from resolv.conf (thus possibly several
> resolvers) which will ensure the exclusivity between the two mechanisms.

Yes, that's a good point. In fact, I don't see why the fallback has to
be implicit.

The confusion often arises because haproxy accepts a resolver
configuration where no resolvers are configured. Maybe we should
reject the configuration when a resolver is referred to in the servers
lines, but no actual resolvers are configured (AND resolv.conf parsing
is not enabled in future).



Lukas

Re: resolvers - resolv.conf fallback

[Willy Tarreau]

Hi guys,

On Wed, Apr 04, 2018 at 12:32:41PM +0200, Lukas Tribus wrote:
> Hello Baptiste,
> 
> 
> > - (for Lukas) what do you think is better, a configuration option to trigger
> > parsing of resolv.conf or as proposed, if no nameserver are found, we use
> > resolv.conf as a failback?
> 
> 
> I don't think we need a config knob for this; currently we don't do
> anything when no nameservers are configured; that behavior combined
> with the by-default enabled libc resolver at startup can cause some
> confusion.
> 
> Transitioning to a resolv.conf fallback is the correct thing to do here imho.
> 
> 
> Just:
> - only fallback if no resolvers are configured in haproxy configuration
> - don't fallback if configured resolvers are unresponsive
> - update the documentation at the same time
> 
> 
> I don't think we need a new config know.

Just thinking, is the goal *not to have to* configure "resolve" on
server lines in this case, or to avoid having to pre-configure the
resolvers themselves when they're the same as the system's ?

If the former, that would mean always enabling DNS resolving at runtime
which doesn't sound like a good idea at all to me. If the latter, then
why not have a special directive in the resolvers section to indicate
that it should use resolv.conf instead ? That could avoid some surprizes
when you simply comment your all your resolvers and that suddenly the
behaviour changes. I'd even say that this directive could serve to
populate the resolvers section from resolv.conf (thus possibly several
resolvers) which will ensure the exclusivity between the two mechanisms.

Cheers,
Willy

Re: resolvers - resolv.conf fallback

[Lukas Tribus]

Hello Baptiste,


> - (for Lukas) what do you think is better, a configuration option to trigger
> parsing of resolv.conf or as proposed, if no nameserver are found, we use
> resolv.conf as a failback?


I don't think we need a config knob for this; currently we don't do
anything when no nameservers are configured; that behavior combined
with the by-default enabled libc resolver at startup can cause some
confusion.

Transitioning to a resolv.conf fallback is the correct thing to do here imho.


Just:
- only fallback if no resolvers are configured in haproxy configuration
- don't fallback if configured resolvers are unresponsive
- update the documentation at the same time


I don't think we need a new config know.



cheers,

lukas

Re: resolvers - resolv.conf fallback

[Baptiste]

On Mon, Apr 2, 2018 at 5:09 PM, Ben Draut  wrote:

> Assuming no one else is already working on the feature, I'd like to
> contribute
> the change to have the resolvers section fall back to parsing
> /etc/resolv.conf
> when no nameserver directives are present.
>
> This was previously discussed here: https://www.mail-archive
> .com/haproxy@formilux.org/msg28626.html.
> (Jim Freeman and I are colleagues.)
>
> I've been surveying the code to see how it could be implemented. I think it
> would make sense to register a 'post_section_parser' for the resolvers
> section
> that would add the nameservers from /etc/resolv.conf if no nameservers were
> specified in the section. As Jim pointed out previously, libresolv could
> be used
> to parse the file. If that's undesirable for some reason, we could parse it
> ourselves.
>
> I'm new to the codebase, so I'm open to any suggestions or guidance anyone
> may have.
>
> Thanks,
>
> Ben
>
>

Hi Ben,

I welcome this feature, though I have a few comments:
- I have nothing against libresolv, but I think this function should be
implemented natively in HAProxy
- (for Lukas) what do you think is better, a configuration option to
trigger parsing of resolv.conf or as proposed, if no nameserver are found,
we use resolv.conf as a failback?

As the maintainer of the DNS code in HAProxy, don't hesitate to ask me any
questions.

Baptiste

resolvers - resolv.conf fallback

[Ben Draut]

Assuming no one else is already working on the feature, I'd like to
contribute
the change to have the resolvers section fall back to parsing
/etc/resolv.conf
when no nameserver directives are present.

This was previously discussed here: https://www.mail-
archive.com/haproxy@formilux.org/msg28626.html.
(Jim Freeman and I are colleagues.)

I've been surveying the code to see how it could be implemented. I think it
would make sense to register a 'post_section_parser' for the resolvers
section
that would add the nameservers from /etc/resolv.conf if no nameservers were
specified in the section. As Jim pointed out previously, libresolv could be
used
to parse the file. If that's undesirable for some reason, we could parse it
ourselves.

I'm new to the codebase, so I'm open to any suggestions or guidance anyone
may have.

Thanks,

Ben