Re: Bind9 on VMWare

[Doug Barton]

On 01/13/2016 04:34 AM, Philippe Maechler wrote:


My idea for the new setup is:
---
caching servers
- Setup new caching servers
- Configure the ipv4 addresses of both (old) servers on the new servers as a
/32 and setup an anycast network.
This way the stupid clients, who won't switch to the secondary ns server
when the primary is not available, are happy when there is some problem with
one server.
If we're having issues with the load in the future we can setup a new server
and put it into the anycast network


Assuming you can manage the anycast effectively that's a good 
architecture that works well. Many of my customers have used it.



auth. servers
- Setup a hidden master on the vmware
- Setup two physical servers which are slaves of the hidden master
That way we have one box which is (anytime in the future) doing the dnssec
stuff, gets the update that we're doing over the webinterface and deploys
the ready-to-serve zones to his slaves.


I would not hesitate to make the authoritative servers virtual as well.


I'm not sure if it is a good thing to have physical serves, although we have
a vmware cluster in both nodes which has enough capacity (ram, cpu, disk)?
I once read that the vmware boxes have a performance issue with heavy udp
based services. Did anyone of you face such an issue? Are your dns servers
all running on physical or virtual boxes?


When I was at BlueCat we recommended to customers that they put their 
resolving name servers on physical boxes in order to avoid chicken and 
egg problems after a catastrophic failure. Resolvers are core 
infrastructure, as are Virtualization clusters. It's better to avoid 
interdependencies between critical infrastructure wherever possible. 
Since you already have the physical boxes, I would continue to use them. 
The same argument can be made for DHCP as well, BTW.


That said, a non-zero number of our customers had all of their stuff 
virtualized, and were quite happy with it. Modern VMware has little or 
no penalty, and certainly nothing that would slow you down at 15k qps.


hope this helps,

Doug

___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: Extracting stats from BIND XML stats file : issues

[John Wobus]

I haven’t used the XML stats so I have no direct experience.
Your formula looks right to me.

If it were me, I’d confirm the load traffic as much as I could,
e.g. os udp counts, tcpdumps at both ends, etc.  I’d
want to make sure it was bind that was doing something strange.

It occurred to me that a bind rrl configuration could affect
this, but a little thought told me that was unlikely to be the
issue.

John Wobus
Cornell IT
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: Bind9 on VMWare

[John Wobus]

Re vmware, I’m definitely interested in anything folks
have discovered about udp performance issues but I have
no negative experience to offer.  We mix vmware and hardware,
but have both auth and query servers on both.  Load tests
didn’t reveal any issues that made us reconsider.

We had an interesting time when we migrated a DNS server that
doubled as our central ntp server into vmware.
Later we moved the ntp server back to bare metal somewhere.
But the issue was not udp; it was the virtualized “hardware” clock.

I have a personal concern about dependencies, e.g. if you ever have
to deal with a problem that’s taken a whole vmware cluster
down.  If the infrastructure or the folks attempting
to fix the infrastructure depend on dns,
or even if they merely work more efficiently when dns is there,
then having that huge single point of failure that takes down
dns could have costs.  Same for a lot of low-level services.
Overall architectures can take this into account.

John Wobus
Cornell University IT
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

RE: DNS BIND traffic capture ICMP/UDP

[Darcy Kevin (FCA)]

I would also recommend looking deeper into the packet capture to determine what 
queries were being responded to. As Warren pointed out, if the initiator times 
out its original query, it will either fail the whole lookup, or at least move 
to another resolver, so in either case the original socket is closed and your 
caching server get an “ICMP Port Unreachable” when it sends its response. But 
this resolution slowness might not be for *everything* that a particular 
caching server tries to resolve; it might only be for certain names. 
Selectively-slow resolution can have multiple causes, but it might be worth 
checking into.

If this were a DoS attack/attempt, then the volume of actual query/response 
traffic – which has a much greater resource impact on your server -- would be 
greater than the ICMP traffic, so it would be pretty obvious that that’s what 
it was. Occasional or sporadic “ICMP Port Unreachable”s are more likely to be 
an unintentional effect of an infrastructure problem, or possibly just bad 
client behavior (e.g. not waiting long enough for the lookup to finish).



- Kevin

From: bind-users-boun...@lists.isc.org 
[mailto:bind-users-boun...@lists.isc.org] On Behalf Of Warren Kumari
Sent: Friday, January 15, 2016 11:32 AM
To: Daniel Dawalibi; bind-users@lists.isc.org
Subject: Re: DNS BIND traffic capture ICMP/UDP


On Fri, Jan 15, 2016 at 8:49 AM Daniel Dawalibi 
mailto:daniel.dawal...@idm.net.lb>> wrote:
Hello

We observed an unusual traffic combining ICMP and UDP packets while running the 
tcpdump command on the DNS caching server
Kindly note that only UDP DNS traffic is allowed on this server (ICMP is not 
allowed from outside to DNS server)
Any help regarding this issue? Why we are getting ICMP and UDP requests? Could 
it be an attack?


Logs:

# tcpdump –n icmp

15:41:05.054237 IP 10.151.130.74 > DNSIP: ICMP 10.151.130.74 udp port 52003 
unreachable, length 52
15:41:05.064449 IP 10.75.6.36 > DNSIP: ICMP 10.75.6.36 udp port 50162 
unreachable, length 52
15:41:05.067953 IP 10.33.10.155 > DNSIP: ICMP 10.33.10.155 udp port 50233 
unreachable, length 52
15:41:05.067958 IP 10.75.15.162 > DNSIP: ICMP 10.75.15.162 udp port 53847 
unreachable, length 52
15:41:05.072727 IP 10.33.12.219 > DNSIP: ICMP 10.33.12.219 udp port 51024 
unreachable, length 52
….
Example: 10.151.130.74 (client source IP)
DNSIP: DNSServer IP


Your description is either incomplete, or incorrect (or at least sine set of 
things is misconfigured) -- without additional information it will be difficult 
/ impossible to assist.

1: You state that you observe traffic while running tcpdump **on the caching 
server**.
2: You state that "ICMP is not allowed from outside **to** DNS server" 
(emphasis mine) - this implies that ICMP is supposed to be filtered before 
reaching the server, not e.g iptables *on the server*.
3: The tcpdump output shows traffic from client IPs (presumably "outside") to 
the DNS server.

I do not see how all of the above can simultaneously be true
A: What are the actual IPs involved?
B: What are you counting as "outside" (are the client IPs "inside" or 
"outside"?)?.
C: Where are you filtering the ICMP, and, more importantly, why are you 
filtering ICMP (it is needed to make IP work properly...)
D: How busy is the server / what percentage of ICMP responses to DNS queries?
E: What is the connectivity of the server? It is likely that resolutions are 
taking significant time, and the clients have a: given up or b: already gotten 
the replies from another recursive?


This could be an attack (e.g spoofed packets as part of a cache poisoning 
attempt) or it could be perfectly normal operation -- eliding the IP 
addresses and not providing more information makes it imposs^W hard to tell

W

Regards
Daniel
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS BIND traffic capture ICMP/UDP

[Warren Kumari]

On Fri, Jan 15, 2016 at 8:49 AM Daniel Dawalibi 
wrote:

> Hello
>
>
>
> We observed an unusual traffic combining ICMP and UDP packets while
> running the tcpdump command on the DNS caching server
>
> Kindly note that only UDP DNS traffic is allowed on this server (ICMP is
> not allowed from outside to DNS server)
>
> Any help regarding this issue? Why we are getting ICMP and UDP requests?
> Could it be an attack?
>
>
>
>
>
> *Logs:*
>
>
>
> # tcpdump –n icmp
>
>
>
> 15:41:05.054237 IP 10.151.130.74 > DNSIP: ICMP 10.151.130.74 udp port
> 52003 unreachable, length 52
>
> 15:41:05.064449 IP 10.75.6.36 > DNSIP: ICMP 10.75.6.36 udp port 50162
> unreachable, length 52
>
> 15:41:05.067953 IP 10.33.10.155 > DNSIP: ICMP 10.33.10.155 udp port 50233
> unreachable, length 52
>
> 15:41:05.067958 IP 10.75.15.162 > DNSIP: ICMP 10.75.15.162 udp port 53847
> unreachable, length 52
>
> 15:41:05.072727 IP 10.33.12.219 > DNSIP: ICMP 10.33.12.219 udp port 51024
> unreachable, length 52
>
> ….
>
> Example: 10.151.130.74 (client source IP)
>
> DNSIP: DNSServer IP
>
>
>

Your description is either incomplete, or incorrect (or at least sine set
of things is misconfigured) -- without additional information it will be
difficult / impossible to assist.

1: You state that you observe traffic while running tcpdump **on the
caching server**.
2: You state that "ICMP is not allowed from outside **to** DNS server"
(emphasis mine) - this implies that ICMP is supposed to be filtered before
reaching the server, not e.g iptables *on the server*.
3: The tcpdump output shows traffic from client IPs (presumably "outside")
to the DNS server.

I do not see how all of the above can simultaneously be true
A: What are the actual IPs involved?
B: What are you counting as "outside" (are the client IPs "inside" or
"outside"?)?.
C: Where are you filtering the ICMP, and, more importantly, why are you
filtering ICMP (it is needed to make IP work properly...)
D: How busy is the server / what percentage of ICMP responses to DNS
queries?
E: What is the connectivity of the server? It is likely that resolutions
are taking significant time, and the clients have a: given up or b: already
gotten the replies from another recursive?


This could be an attack (e.g spoofed packets as part of a cache poisoning
attempt) or it could be perfectly normal operation -- eliding the IP
addresses and not providing more information makes it imposs^W hard to
tell

W


> Regards
>
> Daniel
> ___
> Please visit https://lists.isc.org/mailman/listinfo/bind-users to
> unsubscribe from this list
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS BIND traffic capture ICMP/UDP

[Ray Bellis]

On 15/01/2016 13:48, Daniel Dawalibi wrote:
> Hello
> 
>  
> 
> We observed an unusual traffic combining ICMP and UDP packets while
> running the tcpdump command on the DNS caching server
> 
> Kindly note that only UDP DNS traffic is allowed on this server (ICMP is
> not allowed from outside to DNS server)
> 
> Any help regarding this issue? Why we are getting ICMP and UDP requests?
> Could it be an attack?

The far end is complaining that responses that your server has sent
cannot be delivered to the originator because there's no longer anything
listening at the source port from which the DNS request came.

This could be for several reasons:

1.  the far end's stateful firewall has "timed out" the state it was
maintaining for its own outgoing UDP query

2.  the far end ran out of state memory (similar to 1)

3.  the packet didn't really come from there in the first place (i.e.
the source was spoofed)

Your own firewall is likely permitting the inbound ICMP (despite rules
prohibiting unsolicited inbound ICMP) because as far as it's concerned,
these are *not* unsolicited ICMP packets - they relate to an existing flow.

Ray


___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

RE: Problem in Performance test

[MURTARI, JOHN]

--- Original Msg -
From: "RunxiaWan" 
Subject: Problem in Performance test

Hi all,
I am doing performance test for my company's resolver with BIND 9.10.3 and
find something weird. The test client and resolver are in the same LAN. When
I use a small set of domain as an input with a 1 per second query
sending rate, everything looks reasonable. However, when I use a set of
thousands of domains as an input, The QPS is unexpectedly low and the
latency is high. Here is the result from DNSperf:

  Queries sent: 11823
  Queries completed:11823 (100.00%)
  Queries lost: 0 (0.00%)

  Response codes:   NOERROR 9883 (83.59%), SERVFAIL 242 (2.05%),
NXDOMAIN 1698 (14.36%)
  Average packet size:  request 48, response 203
  Run time (s): 69.891567
  Queries per second:   169.162039

  Average Latency (s):  0.519502 (min 0.003766, max 211.981919)
  Latency StdDev (s):   1.423057

And when I decreased the query sending rate to 100 per second, the latency
decrease as the same when I use small set of domain as an input. Here is the
result from DNSperf:

Statistics:

  Queries sent: 6000
  Queries completed:6000 (100.00%)
  Queries lost: 0 (0.00%)

  Response codes:   NOERROR 4995 (83.25%), SERVFAIL 37 (0.62%), NXDOMAIN
968 (16.13%)
  Average packet size:  request 54, response 211
  Run time (s): 62.789257
  Queries per second:   95.557748

  Average Latency (s):  0.083028 (min 0.005266, max 134.543920)
  Latency StdDev (s):   0.568863

Anyone knows any explanation for this? Thanks.
---
Runxia Wan(Brian)
Research Engineer
BII Lab
Beijing Internet Institute(BII)
rx...@biigroup.cn
---

Few  things are affecting the resolver test:
1) A resolver is allowed to cache answers, in your small test of zones  it may 
have done a series of iterative queries to find the answers, but once it had 
the answers, future queries were answered from cache -- easy going and high 
QPS, just limited by your memory/cpu performance.

2) When you added thousands of random sites it took it much longer to perform 
all the look ups and begin storing cached results.  A much more realistic test, 
but slower going.

3) Not sure if you stopped/started named after each test?  If you just left it 
running, it still had many results cached and could use those in subsequent 
tests.

4) UDP queue length - a realistic test would have several clients running 
dnsperf and querying the same resolver.  As the resolver falls behind, the UDP 
queue of waiting requests grows and some of those are just dropped by the OS.

There's a lot going on under the hood!
Hope this helps!
John






-- next part --
An HTML attachment was scrubbed...
URL: 
<https://lists.isc.org/pipermail/bind-users/attachments/20160115/f9cc36a7/attachment-0001.html>

--

___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

End of bind-users Digest, Vol 2288, Issue 1
***
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

DNS BIND traffic capture ICMP/UDP

[Daniel Dawalibi]

Hello

 

We observed an unusual traffic combining ICMP and UDP packets while running
the tcpdump command on the DNS caching server 

Kindly note that only UDP DNS traffic is allowed on this server (ICMP is not
allowed from outside to DNS server)

Any help regarding this issue? Why we are getting ICMP and UDP requests?
Could it be an attack?

 

 

Logs:

 

# tcpdump -n icmp

 

15:41:05.054237 IP 10.151.130.74 > DNSIP: ICMP 10.151.130.74 udp port 52003
unreachable, length 52

15:41:05.064449 IP 10.75.6.36 > DNSIP: ICMP 10.75.6.36 udp port 50162
unreachable, length 52

15:41:05.067953 IP 10.33.10.155 > DNSIP: ICMP 10.33.10.155 udp port 50233
unreachable, length 52

15:41:05.067958 IP 10.75.15.162 > DNSIP: ICMP 10.75.15.162 udp port 53847
unreachable, length 52

15:41:05.072727 IP 10.33.12.219 > DNSIP: ICMP 10.33.12.219 udp port 51024
unreachable, length 52

..

Example: 10.151.130.74 (client source IP)

DNSIP: DNSServer IP

 

Regards

Daniel

___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: Problem in Performance test

[Tony Finch]

RunxiaWan  wrote:

> However, when I use a set of thousands of domains as an input, The QPS
> is unexpectedly low and the latency is high.

That would be normal if the cache is empty. Did you pre-populate it before
running the benchmark?

> And when I decreased the query sending rate to 100 per second, the latency
> decrease as the same when I use small set of domain as an input.

If you did that after running the previous benchmark, it is benefiting
from a populated cache.

Tony.
-- 
f.anthony.n.finchhttp://dotat.at/
Tyne, Dogger: Northwest 4 or 5, becoming cyclonic 6 to gale 8, occasionally
severe gale 9 in Dogger. Moderate, becoming rough or very rough. Wintry
showers. Good, occasionally poor.
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users