Hi Edo,
I agree that KIP-235 and KIP-302 address different scenarios. And I agree
that each one is not sufficient in itself to address both the scenarios.
But I also think that they conflict and hence they need to be looked at
together and perhaps use a single config.
As an example:
If I run:
for (InetAddress address : InetAddress.getAllByName("www.apache.org")) {
System.out.printf("HostName %s canonicalHostName %s IP %s\n",
address.getHostName(), address.getCanonicalHostName(),
address.getHostAddress());
}
I get:
HostName www.apache.org canonicalHostName tlp-eu-west.apache.org IP
195.154.151.36
HostName www.apache.org canonicalHostName 40.79.78.1 IP 40.79.78.1
HostName www.apache.org canonicalHostName themis.apache.org IP
140.211.11.105
HostName www.apache.org canonicalHostName 2001:bc8:2142:300:0:0:0:0 IP
2001:bc8:2142:300:0:0:0:0
If www.apache.org is used as a bootstrap address, KIP-302 would connect to (
www.apache.org/195.154.151.36 and www.apache.org/140.211.11.105) while
KIP-235 would connect to (tlp-eu-west.apache.org/195.154.151.3. and
themis.apache.org/140.211.11.105). This is a significant difference not
just for Kerberos, but for any secure environment where hostname is
verified to prevent man-in-the-middle attacks. In your case, I presume you
would have SSL certificates with the equivalent of www.apache.org on both
the load balancers. In Jonathan's case, I presume he has Kerberos
principals for the equivalent of tlp-eu-west.apache.org and
themis.apache.org. We would want to support both scenarios regardless of
the security protocol, just need to come up with configuration options that
don't conflict.
On Mon, May 14, 2018 at 5:24 PM, Edoardo Comar <edoco...@gmail.com> wrote:
> Thanks Rajini
>
> I still don't see the overlap between the two KIPS
>
> KIP-235 allows an expansion of hostnames on the bootstrap list.
>
> KIP-302 allows alternative IPs to be used to attempt a connection
> (either at bootstrap and when processing the MetadataResponse) to a
> given hostname.
>
> A use case would be that of active/passive LB fronting the brokers.
>
> Arguably, if Java honored the DNS-set TTL, and the TTL was low and on
> subsequent requests, the order of IPs returned by the
> InetAddress.getAllByName() was random, we may not need such an
> enhancement.
> In practice, a Java client can get stuck on a "bad" IP forever if it
> only relies on the first IP returned.
>
> HTH,
> Edo
>
> On 14 May 2018 at 16:23, Rajini Sivaram <rajinisiva...@gmail.com> wrote:
> > Hi Edo,
> >
> > Thanks for the KIP. I think it will be good to include a diagram to make
> it
> > easier to distinguish this scenario from that of KIP-235 without reading
> > the PR.
> >
> > It may be worth considering if KIP-235 and this KIP could use a single
> > config name with different values instead of two boolean configs:
> >
> > bootstrap.reverse.dns.lookup = true/false
> > enable.all.dns.ips = true/false
> >
> > Not all values of (bootstrap.reverse.dns.lookup, enable.all.dns.ips) seem
> > to make sense. And not all scenarios are handled. Even if we use multiple
> > configs, it seems to me that we may want to name them differently.
> >
> > The possible combinations are:
> >
> > 1) Bootstrap
> >
> > a) No lookup
> > b) Use all DNS entries with host name
> > c) Use all DNS entries with canonical host name
> >
> > 2) Advertised listeners
> >
> > a) No lookup
> > b) Use all DNS entries with host name
> > c) Use all DNS entries with canonical host name
> >
> > The combinations that are enabled by the two boolean configs (
> > bootstrap.reverse.dns.lookup, enable.all.dns.ips) are:
> >
> > - (false, false) => (1a, 2a)
> > - (true, false) => (1c, 2a)
> > - (false, true) => (1b, 2b)
> > - (true, true) => (??, 2b)
> >
> > It will be good if we can clearly identify which combinations we want to
> > support and the scenarios where they may be useful. Perhaps (1a, 2a),
> (1c,
> > 2a), (1b, 2b) and (1c, 2c) are useful?
> >
> >
> > On Mon, May 14, 2018 at 2:58 PM, Skrzypek, Jonathan <
> > jonathan.skrzy...@gs.com> wrote:
> >
> >> Ah, apologies didn't see there was already a decent amount of discussion
> >> on this in the PR.
> >>
> >> This kind of sounds related to the environment you're running to me.
> >> What is the rationale behind using the advertised listeners to do your
> >> load balancing advertisement rather than a top level alias that has
> >> everything ?
> >>
> >> It sounds like in your case there is a mismatch between
> bootstrap.servers
> >> and advertised.listeners, and you want advertised.listeners to take
> >> precedence and have the client iterate over what is returned by the
> broker.
> >> So the extra parameter doesn't only have to do with DNS but it's also
> >> appending from the broker, maybe the parameter name should reflect this
> ?
> >>
> >> Jonathan Skrzypek
> >>
> >>
> >> -----Original Message-----
> >> From: Skrzypek, Jonathan [Tech]
> >> Sent: 14 May 2018 14:46
> >> To: dev@kafka.apache.org
> >> Subject: RE: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> >> resolved IP addresses
> >>
> >> Hi,
> >>
> >> I see you noted the similarities with KIP-235.
> >> But KIP-235 might also solve what this KIP is trying to achieve.
> >>
> >> When parsing bootstrap.servers, KIP-235 has the client add all
> underlying
> >> hostnames and IPs.
> >> And this happens before hitting the NetworkClient.
> >>
> >> So to me the client will try every single endpoint behind any
> >> bootstrap.servers record.
> >>
> >> See https://github.com/apache/kafka/pull/4485/commits/24757eb7b0
> >> 6bcf8c7d7649c85232c52b5d54f0e4#diff-89ef153462e64c250a21bd324ae1a851
> >> which calls getAllByName like you suggested
> >>
> >> Jonathan Skrzypek
> >>
> >>
> >> -----Original Message-----
> >> From: Edoardo Comar [mailto:edoco...@gmail.com]
> >> Sent: 14 May 2018 14:17
> >> To: dev@kafka.apache.org
> >> Subject: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> resolved
> >> IP addresses
> >>
> >> Hi all,
> >>
> >> We just opened a KIP to add support for the client to use all IPs
> returned
> >> by DNS for the brokers
> >>
> >> The details are here -
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.a
> >> pache.org_confluence_display_KAFKA_KIP-2D302-2B-2D-2BEnable-
> >> 2BKafka-2Bclients-2Bto-2Buse-2Ball-2BDNS-2Bresolved-2BIP-
> >> 2Baddresses&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxL
> >> xfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=EJafF
> >> l1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&s=C-UZ6KUG7JFiPD_
> >> CnHczDOVqH9-XC5f_OFkw4BTNrI4&e=
> >>
> >> The JIRA and provisional PR (where the discussion lead to the creation
> of
> >> this KIP) are :
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> >> apache.org_jira_browse_KAFKA-2D6863&d=DwIBaQ&c=7563p3e2zaQw0
> >> AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6
> >> eaCbPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-
> >> D2Dti4k&s=3Puqs5iYoPsw6hARQr6gvokdFE-H5USMiNVGOUtNkJI&e=
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> >> com_apache_kafka_pull_4987&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgy
> >> agb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaC
> >> bPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&
> >> s=Hqn5dOgQy4-MHTIJLE49O8bNomry3SoGq9OVoHU-CRA&e=
> >>
> >> Looking forward to the community's feedback.
> >> It would be amazing to have it voted by May 22nd :-) :-)
> >>
> >> Edoardo & Mickael
> >>
>
>
>
> --
> "When the people fear their government, there is tyranny; when the
> government fears the people, there is liberty." [Thomas Jefferson]
>
