Re: DNS traffic tracking

[Alex K]

On Mon, May 9, 2022 at 7:27 PM Fred Morris  wrote:

> On Mon, 9 May 2022, Alex K wrote:
> > [...]
> > The problem now is that I see sometime 700MB of DNS traffic for 2GB of
> > Internet browsing within one month.
>
> That's an eyebrow raiser. Tunneling, antivirus (or some other database
> using DNS as a key+value store), CDN? IoT fleet? Then comes the inevitable
> "...or traffic you don't want".
>
> Not clear on where the expensive link sits (between the caching resolver
> and clients, or between the caching resolver and the rest of the
> internet). Not sure what you're able to do politically or where things
> like privacy or "net neutrality" come into play, but it does seem to me
> that not burning precious bandwidth for ads might be a value-added
> service... if they're really watching cat videos.
>
The setup is edge device where a caching DNS server runs and where the
users are serviced -> satellite -> upstream DNS servers that can be either
public ones or my second level of caching DNS server depending on the
setup.  The expensive link is from the edge device to the next hop which is
through satellite, and depending on the satellite type may have low
allowance on the monthly traffic (4GB to 8GB max)

>
> I second the comment that Dnstap might be your best friend.
>
> There are technical considerations, but I think generally this is veering
> into the realm of what's possible (which is seldom actually technical);
> this includes your means and ability to analyze the DNS traffic. If you
> want to discuss further feel free to email me.
>
> Thanx for all the feedback. I will need to drill down and see what kind of
DNS traffic is that then perhaps implement some more secure firewalling
(find a way to block VPN over DNS) and rate limiting.
I was also thinking perhaps to have a preloaded RPZ list that will block
malware domains at the caching DNS server at the edge.

> --
>
> Fred Morris, internet plumber
>
> --
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
> from this list
>
> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
>
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Fred Morris]

On Mon, 9 May 2022, Alex K wrote:

[...]
The problem now is that I see sometime 700MB of DNS traffic for 2GB of
Internet browsing within one month.


That's an eyebrow raiser. Tunneling, antivirus (or some other database 
using DNS as a key+value store), CDN? IoT fleet? Then comes the inevitable 
"...or traffic you don't want".


Not clear on where the expensive link sits (between the caching resolver 
and clients, or between the caching resolver and the rest of the 
internet). Not sure what you're able to do politically or where things 
like privacy or "net neutrality" come into play, but it does seem to me 
that not burning precious bandwidth for ads might be a value-added 
service... if they're really watching cat videos.


I second the comment that Dnstap might be your best friend.

There are technical considerations, but I think generally this is veering 
into the realm of what's possible (which is seldom actually technical); 
this includes your means and ability to analyze the DNS traffic. If you 
want to discuss further feel free to email me.


--

Fred Morris, internet plumber

--
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Peter Coghlan]

Alex K wrote:
>On Mon, May 9, 2022 at 2:46 PM Bjørn Mork  wrote:
>>
>> FWIW I agree with the rate-limit recommendation.  It solves both this
>> and your original problem without any complicated and messy tracking.
>> Just make DNS "free" up to some reasonable query rate.  If there are
>> clients with higher legitimate needs, then you could consider creating
>> separate rate-limit classes for those clients.  And even charge extra
>> for that, if it's important.
>>
> Does such DNS traffic has different characteristics from the normal one?
> Perhaps, apart from limiting, I could block such traffic with the packet
> size or similar.
>

In a different message you said "I see sometime 700MB of DNS traffic for
2GB of Internet browsing within one month."  This seems like a huge figure
and it suggests that something very undesirable is happening somewhere.

If you are counting traffic between your client's resolvers and their
nearest caching nameserver, perhaps adding caching nameservers closer to
or at your client's sites would be helpful?

Maybe someone is attacking your clients with lots of bogus DNS queries?

Maybe some of your clients have been infected with malware and are
attacking the rest of the world with lots of bogus DNS queries?

Someone else suggested your clients could be doing IP over DNS.

Maybe something else we haven't thought of is happening?

Rather than speculating as to what is the cause of this huge figure
for DNS traffic, or trying to generate and analyse logs which may or
may not cover the traffic in question, I would suggest doing some
packet sniffing with tcpdump or wireshark or whatever works in your
environment and figuring out exactly what the traffic is and getting
a better idea of who is responsible for generating it and why.

In my opinion, in the absence of knowing what the problem is,
experimenting with stuff like rate limiting or blocking is unlikely
to solve the problem.

Regards,
Peter Coghlan.
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Alex K]

On Mon, May 9, 2022 at 2:46 PM Bjørn Mork  wrote:

> Alex K  writes:
> > On Mon, May 9, 2022 at 1:51 PM Matus UHLAR - fantomas  >
> > wrote:
> >
> >> maybe someone uses VPN over DNS...
> >> in such case, rate limiting of client comes to mind...
> >>
> > That would mean that the clients have access to their own dns servers,
> > which the firewall does not allow.
>
> No, you can run IP over DNS using any resolver.  Also yours.
>
> Yes, they need a server for the remote end. But your resolver will be
> the one talking to it, just like it queries any other autoritative
> server on behalf of the client.
>
> Typically something you do for fun. Not for normal use.  But I guess it
> could be in use by those who need a reliable communication channel
> inside any "isolated" environment.  DNS tends to be availble even where
> nothing else is.
>
I see. thanx for clarifying.


>
> FWIW I agree with the rate-limit recommendation.  It solves both this
> and your original problem without any complicated and messy tracking.
> Just make DNS "free" up to some reasonable query rate.  If there are
> clients with higher legitimate needs, then you could consider creating
> separate rate-limit classes for those clients.  And even charge extra
> for that, if it's important.
>
Does such DNS traffic has different characteristics from the normal one?
Perhaps, apart from limiting, I could block such traffic with the packet
size or similar.


>
> Bjørn
> --
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
> from this list
>
> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
>
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Bjørn Mork]

Alex K  writes:
> On Mon, May 9, 2022 at 1:51 PM Matus UHLAR - fantomas 
> wrote:
>
>> maybe someone uses VPN over DNS...
>> in such case, rate limiting of client comes to mind...
>>
> That would mean that the clients have access to their own dns servers,
> which the firewall does not allow.

No, you can run IP over DNS using any resolver.  Also yours.

Yes, they need a server for the remote end. But your resolver will be
the one talking to it, just like it queries any other autoritative
server on behalf of the client.

Typically something you do for fun. Not for normal use.  But I guess it
could be in use by those who need a reliable communication channel
inside any "isolated" environment.  DNS tends to be availble even where
nothing else is.

FWIW I agree with the rate-limit recommendation.  It solves both this
and your original problem without any complicated and messy tracking.
Just make DNS "free" up to some reasonable query rate.  If there are
clients with higher legitimate needs, then you could consider creating
separate rate-limit classes for those clients.  And even charge extra
for that, if it's important.


Bjørn
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Alex K]

On Mon, May 9, 2022 at 1:51 PM Matus UHLAR - fantomas 
wrote:

> >On 09. 05. 22 10:34, Alex K wrote:
> >>The initial and current approach is to provide DNS free of charge,
> >>which simplified things for me. Though the traffic in question is
> >>satellite traffic with monthly allowances of roughly 4 to 8GB, thus
> >>every MB counts.
> >>The problem now is that I see sometime 700MB of DNS traffic for 2GB
> >>of Internet browsing within one month.
>
> On 09.05.22 10:47, Petr Špaček wrote:
> >Sounds like either:
> >- Broken caching or,
> >- Random subdomain attack
> >to me.
>
> maybe someone uses VPN over DNS...
> in such case, rate limiting of client comes to mind...
>
That would mean that the clients have access to their own dns servers,
which the firewall does not allow.


> --
> Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> Support bacteria - they're the only culture some people have.
> --
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
> from this list
>
> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
>
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Matus UHLAR - fantomas]

On 09. 05. 22 10:34, Alex K wrote:
The initial and current approach is to provide DNS free of charge, 
which simplified things for me. Though the traffic in question is 
satellite traffic with monthly allowances of roughly 4 to 8GB, thus 
every MB counts.
The problem now is that I see sometime 700MB of DNS traffic for 2GB 
of Internet browsing within one month.


On 09.05.22 10:47, Petr Špaček wrote:

Sounds like either:
- Broken caching or,
- Random subdomain attack
to me.


maybe someone uses VPN over DNS...
in such case, rate limiting of client comes to mind...

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Support bacteria - they're the only culture some people have.
--
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Petr Špaček]

On 09. 05. 22 12:06, Alex K wrote:

Hi Greg,

On Mon, May 9, 2022 at 11:17 AM Greg Choules 
> wrote:


Hi Alex.
Your use case may be very different to the one I faced in my
previous job. But there we did not and could not charge for DNS. It
was seen as a necessary but free resource.
If you *really* want to account for how many queries clients are
making, a quick and dirty solution is enabling querylog, BUT be
warned it causes a lot more load on the system. The better tool
would be DNStap.

I would rather prefer to avoid enabling query logs. One other thing I 
was thining is to just see if bind9 logs the cache hit ratio in the 
stats and use that as as rough coefficient for the internal client 
traffic accounting.


There is bunch of data available in the statistics channel:
https://bind9.readthedocs.io/en/latest/reference.html#statistics-counters

Beware:
It might give you only a very rough estimate, like, "is cache hit rate 
on average 0, 1/10, 1/2, or 9/10".


It is good enough to detect that a client engaged in a random subdomain 
attacks and you need to look into traffic, but that's about it.


--
Petr Špaček
--
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Alex K]

Hi Greg,

On Mon, May 9, 2022 at 11:17 AM Greg Choules <
gregchoules+bindus...@googlemail.com> wrote:

> Hi Alex.
> Your use case may be very different to the one I faced in my previous job.
> But there we did not and could not charge for DNS. It was seen as a
> necessary but free resource.
> If you *really* want to account for how many queries clients are making,
> a quick and dirty solution is enabling querylog, BUT be warned it causes a
> lot more load on the system. The better tool would be DNStap.
>
I would rather prefer to avoid enabling query logs. One other thing I was
thining is to just see if bind9 logs the cache hit ratio in the stats and
use that as as rough coefficient for the internal client traffic
accounting.


>
> But there is no 1-to-1 correlation between user queries (client side of
> server) and fetches (Internet side of server).
> In a perfect (i.e. lab) setup, if all clients make the same query then,
> apart from the initial fetches to find the answer(s) the server can answer
> everything from cache and there is no internet traffic at all. (100% cache
> hit ratio)
> The other extreme is clients all making random queries (PRSD), which your
> server cannot cache, so this causes it to generate much more Internet
> traffic; at least as much as the clients are generating. (0% cache hit
> ratio)
>
> Cheers, Greg
>
>
>
> On Fri, 6 May 2022 at 16:02, Alex K  wrote:
>
>> Hi all,
>>
>> I have the following problem: I run a caching dns server using bind9
>> v9.10.3 in a gateway device which it serves several internal LAN IP
>> addresses (clients). I am doing some traffic accounting in the gateway
>> device using Linux conntrack so as to calculate the generated client
>> traffic (mostly HTTP/HTTPs related, in/out) so as to charge the volume
>> consumed.
>>
>> What I cannot charge is the actual DNS traffic that each client is
>> generating, since each client DNS request is actually two sessions, one
>> between client and gateway device and the other between gateway and
>> upstream DNS servers. It seems to me not fare to charge the traffic
>> observed between the client and the gateway since the internal DNS traffic
>> includes cached responses and may be much higher from the actual DNS
>> traffic observed on the WAN side (gateway - upstream).
>>
>> I was wondering if there is a solution to this. If bind9 has any feature
>> that can be used to track the WAN DNS traffic and understand from which
>> client was first requested/generated. In this way I will be able to
>> differentiate the DNS traffic per client and avoid accounting DNS traffic
>> that the gateway generated for its own services.
>>
>> Just as an additional note on this, I had in the past the same issue with
>> the proxy traffic that this same gateway was generating and found a
>> solution by using TPROXY feature of the squid proxy, which exposes the real
>> internal client IP address at the WAN traffic which can later be NATed.
>>
>> Thanx for any ideas,
>> Alex
>> --
>> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
>> from this list
>>
>> ISC funds the development of this software with paid support
>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>> information.
>>
>>
>> bind-users mailing list
>> bind-users@lists.isc.org
>> https://lists.isc.org/mailman/listinfo/bind-users
>>
>
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Alex K]

On Mon, May 9, 2022 at 11:48 AM Petr Špaček  wrote:

> On 09. 05. 22 10:34, Alex K wrote:
> > Hi Petr,
> >
> > On Mon, May 9, 2022 at 10:26 AM Petr Špaček  > > wrote:
> >
> > On 06. 05. 22 17:02, Alex K wrote:
> >  > Hi all,
> >  >
> >  > I have the following problem: I run a caching dns server using
> bind9
> >  > v9.10.3 in a gateway device which it serves several internal LAN
> IP
> >  > addresses (clients). I am doing some traffic accounting in the
> > gateway
> >  > device using Linux conntrack so as to calculate the generated
> client
> >  > traffic (mostly HTTP/HTTPs related, in/out) so as to charge the
> > volume
> >  > consumed.
> >  >
> >  > What I cannot charge is the actual DNS traffic that each client is
> >  > generating, since each client DNS request is actually two
> > sessions, one
> >  > between client and gateway device and the other between gateway
> and
> >  > upstream DNS servers. It seems to me not fare to charge the
> traffic
> >  > observed between the client and the gateway since the internal DNS
> >  > traffic includes cached responses and may be much higher from the
> > actual
> >  > DNS traffic observed on the WAN side (gateway - upstream).
> >  >
> >  > I was wondering if there is a solution to this. If bind9 has any
> > feature
> >  > that can be used to track the WAN DNS traffic and understand from
> > which
> >  > client was first requested/generated. In this way I will be able
> to
> >  > differentiate the DNS traffic per client and avoid accounting DNS
> >  > traffic that the gateway generated for its own services.
> >
> > It cannot be done because there is no 1:1 mapping between client and
> > authoritative side of BIND. Multiple client queries might be solved
> > by a
> > single query to authoritative side, or a single query might cause
> > multiple interrelated queries.
> >
> > If money are involved then I say "don't even try": All reasonable
> > solutions will cause either overcharging or undercharging, which is
> not
> > only objectionable but also possibly illegal.
> >
> > Out of curiosity, is the amount of traffic so large it is worth
> > considering it? Compared to all the YouTube videos? :-)
> >
> > The initial and current approach is to provide DNS free of charge, which
> > simplified things for me. Though the traffic in question is satellite
> > traffic with monthly allowances of roughly 4 to 8GB, thus every MB
> counts.
> > The problem now is that I see sometime 700MB of DNS traffic for 2GB of
> > Internet browsing within one month.
>
> Sounds like either:
> - Broken caching or,
> - Random subdomain attack
> to me.
>
>  >Currently I do not monitor what is
> > the internal/cached DNS vs external/actual DNS traffic so as to know the
> > ratio but it can be significant for such types of deployments. For
> > deployments where the monthly allowance is unlimited no-one ever came to
> > me to ask why DNS is not charged but in this case the customers will
> > need to know where the MBs are consumed. Hope that this clarifies the
> > situation.
> >
> > What I was thinking, as per Josh feedback, is to use ECS and try to find
> > out a way to match that WAN/actual DNS traffic which is initially
> > generated from clients. Then I could use some math to calculate the per
> > client DNS traffic to account, but it's a bit hackish and I cannot think
> > of anything else. The other approach would be to just charge all the
> > internal traffic with the risk of overcharging, as long the the
> > customers agree with it.
>
> ECS with full client identifier is a terrible idea because:
> - It will expose all client IP addresses to rest of the Internet.
> - Is not even allowed by ECS RFCs.
> - It will lower cache hit ratio and you will end up using much more
> traffic for DNS than without ECS.
>
I have two levels of recursive servers due to the current design thus the
final exposed traffic will not include the internal client IP addresses,
but I agree, I would like to avoid ECS since I do not have the required
subscription and would prefer a more simple approach.

>
> (All this this assumes you even have access to BIND ECS support is only
> in the BIND subscription edition.)
>
> I recommend just not going there, do something on resolver-client side
> instead.

Thanx for your feedback. Appreciated.

>
> --
> Petr Špaček
> --
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
> from this list
>
> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
>
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with 

Re: DNS traffic tracking

[Petr Špaček]

On 09. 05. 22 10:34, Alex K wrote:

Hi Petr,

On Mon, May 9, 2022 at 10:26 AM Petr Špaček > wrote:


On 06. 05. 22 17:02, Alex K wrote:
 > Hi all,
 >
 > I have the following problem: I run a caching dns server using bind9
 > v9.10.3 in a gateway device which it serves several internal LAN IP
 > addresses (clients). I am doing some traffic accounting in the
gateway
 > device using Linux conntrack so as to calculate the generated client
 > traffic (mostly HTTP/HTTPs related, in/out) so as to charge the
volume
 > consumed.
 >
 > What I cannot charge is the actual DNS traffic that each client is
 > generating, since each client DNS request is actually two
sessions, one
 > between client and gateway device and the other between gateway and
 > upstream DNS servers. It seems to me not fare to charge the traffic
 > observed between the client and the gateway since the internal DNS
 > traffic includes cached responses and may be much higher from the
actual
 > DNS traffic observed on the WAN side (gateway - upstream).
 >
 > I was wondering if there is a solution to this. If bind9 has any
feature
 > that can be used to track the WAN DNS traffic and understand from
which
 > client was first requested/generated. In this way I will be able to
 > differentiate the DNS traffic per client and avoid accounting DNS
 > traffic that the gateway generated for its own services.

It cannot be done because there is no 1:1 mapping between client and
authoritative side of BIND. Multiple client queries might be solved
by a
single query to authoritative side, or a single query might cause
multiple interrelated queries.

If money are involved then I say "don't even try": All reasonable
solutions will cause either overcharging or undercharging, which is not
only objectionable but also possibly illegal.

Out of curiosity, is the amount of traffic so large it is worth
considering it? Compared to all the YouTube videos? :-)

The initial and current approach is to provide DNS free of charge, which 
simplified things for me. Though the traffic in question is satellite 
traffic with monthly allowances of roughly 4 to 8GB, thus every MB counts.
The problem now is that I see sometime 700MB of DNS traffic for 2GB of 
Internet browsing within one month. 


Sounds like either:
- Broken caching or,
- Random subdomain attack
to me.

>Currently I do not monitor what is
the internal/cached DNS vs external/actual DNS traffic so as to know the 
ratio but it can be significant for such types of deployments. For 
deployments where the monthly allowance is unlimited no-one ever came to 
me to ask why DNS is not charged but in this case the customers will 
need to know where the MBs are consumed. Hope that this clarifies the 
situation.


What I was thinking, as per Josh feedback, is to use ECS and try to find 
out a way to match that WAN/actual DNS traffic which is initially 
generated from clients. Then I could use some math to calculate the per 
client DNS traffic to account, but it's a bit hackish and I cannot think 
of anything else. The other approach would be to just charge all the 
internal traffic with the risk of overcharging, as long the the 
customers agree with it.


ECS with full client identifier is a terrible idea because:
- It will expose all client IP addresses to rest of the Internet.
- Is not even allowed by ECS RFCs.
- It will lower cache hit ratio and you will end up using much more 
traffic for DNS than without ECS.


(All this this assumes you even have access to BIND ECS support is only 
in the BIND subscription edition.)


I recommend just not going there, do something on resolver-client side 
instead.


--
Petr Špaček
--
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Alex K]

Hi Petr,

On Mon, May 9, 2022 at 10:26 AM Petr Špaček  wrote:

> On 06. 05. 22 17:02, Alex K wrote:
> > Hi all,
> >
> > I have the following problem: I run a caching dns server using bind9
> > v9.10.3 in a gateway device which it serves several internal LAN IP
> > addresses (clients). I am doing some traffic accounting in the gateway
> > device using Linux conntrack so as to calculate the generated client
> > traffic (mostly HTTP/HTTPs related, in/out) so as to charge the volume
> > consumed.
> >
> > What I cannot charge is the actual DNS traffic that each client is
> > generating, since each client DNS request is actually two sessions, one
> > between client and gateway device and the other between gateway and
> > upstream DNS servers. It seems to me not fare to charge the traffic
> > observed between the client and the gateway since the internal DNS
> > traffic includes cached responses and may be much higher from the actual
> > DNS traffic observed on the WAN side (gateway - upstream).
> >
> > I was wondering if there is a solution to this. If bind9 has any feature
> > that can be used to track the WAN DNS traffic and understand from which
> > client was first requested/generated. In this way I will be able to
> > differentiate the DNS traffic per client and avoid accounting DNS
> > traffic that the gateway generated for its own services.
>
> It cannot be done because there is no 1:1 mapping between client and
> authoritative side of BIND. Multiple client queries might be solved by a
> single query to authoritative side, or a single query might cause
> multiple interrelated queries.
>
> If money are involved then I say "don't even try": All reasonable
> solutions will cause either overcharging or undercharging, which is not
> only objectionable but also possibly illegal.
>
> Out of curiosity, is the amount of traffic so large it is worth
> considering it? Compared to all the YouTube videos? :-)
>
The initial and current approach is to provide DNS free of charge, which
simplified things for me. Though the traffic in question is satellite
traffic with monthly allowances of roughly 4 to 8GB, thus every MB counts.
The problem now is that I see sometime 700MB of DNS traffic for 2GB of
Internet browsing within one month. Currently I do not monitor what is the
internal/cached DNS vs external/actual DNS traffic so as to know the ratio
but it can be significant for such types of deployments. For deployments
where the monthly allowance is unlimited no-one ever came to me to ask why
DNS is not charged but in this case the customers will need to know where
the MBs are consumed. Hope that this clarifies the situation.

What I was thinking, as per Josh feedback, is to use ECS and try to find
out a way to match that WAN/actual DNS traffic which is initially generated
from clients. Then I could use some math to calculate the per client DNS
traffic to account, but it's a bit hackish and I cannot think of anything
else. The other approach would be to just charge all the internal traffic
with the risk of overcharging, as long the the customers agree with it.

>
> --
> Petr Špaček
> --
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
> from this list
>
> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
>
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Greg Choules via bind-users]

Hi Alex.
Your use case may be very different to the one I faced in my previous job.
But there we did not and could not charge for DNS. It was seen as a
necessary but free resource.
If you *really* want to account for how many queries clients are making,
a quick and dirty solution is enabling querylog, BUT be warned it causes a
lot more load on the system. The better tool would be DNStap.

But there is no 1-to-1 correlation between user queries (client side of
server) and fetches (Internet side of server).
In a perfect (i.e. lab) setup, if all clients make the same query then,
apart from the initial fetches to find the answer(s) the server can answer
everything from cache and there is no internet traffic at all. (100% cache
hit ratio)
The other extreme is clients all making random queries (PRSD), which your
server cannot cache, so this causes it to generate much more Internet
traffic; at least as much as the clients are generating. (0% cache hit
ratio)

Cheers, Greg



On Fri, 6 May 2022 at 16:02, Alex K  wrote:

> Hi all,
>
> I have the following problem: I run a caching dns server using bind9
> v9.10.3 in a gateway device which it serves several internal LAN IP
> addresses (clients). I am doing some traffic accounting in the gateway
> device using Linux conntrack so as to calculate the generated client
> traffic (mostly HTTP/HTTPs related, in/out) so as to charge the volume
> consumed.
>
> What I cannot charge is the actual DNS traffic that each client is
> generating, since each client DNS request is actually two sessions, one
> between client and gateway device and the other between gateway and
> upstream DNS servers. It seems to me not fare to charge the traffic
> observed between the client and the gateway since the internal DNS traffic
> includes cached responses and may be much higher from the actual DNS
> traffic observed on the WAN side (gateway - upstream).
>
> I was wondering if there is a solution to this. If bind9 has any feature
> that can be used to track the WAN DNS traffic and understand from which
> client was first requested/generated. In this way I will be able to
> differentiate the DNS traffic per client and avoid accounting DNS traffic
> that the gateway generated for its own services.
>
> Just as an additional note on this, I had in the past the same issue with
> the proxy traffic that this same gateway was generating and found a
> solution by using TPROXY feature of the squid proxy, which exposes the real
> internal client IP address at the WAN traffic which can later be NATed.
>
> Thanx for any ideas,
> Alex
> --
> Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
> from this list
>
> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
>
>
> bind-users mailing list
> bind-users@lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
>
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

Re: DNS traffic tracking

[Petr Špaček]

On 06. 05. 22 17:02, Alex K wrote:

Hi all,

I have the following problem: I run a caching dns server using bind9 
v9.10.3 in a gateway device which it serves several internal LAN IP 
addresses (clients). I am doing some traffic accounting in the gateway 
device using Linux conntrack so as to calculate the generated client 
traffic (mostly HTTP/HTTPs related, in/out) so as to charge the volume 
consumed.


What I cannot charge is the actual DNS traffic that each client is 
generating, since each client DNS request is actually two sessions, one 
between client and gateway device and the other between gateway and 
upstream DNS servers. It seems to me not fare to charge the traffic 
observed between the client and the gateway since the internal DNS 
traffic includes cached responses and may be much higher from the actual 
DNS traffic observed on the WAN side (gateway - upstream).


I was wondering if there is a solution to this. If bind9 has any feature 
that can be used to track the WAN DNS traffic and understand from which 
client was first requested/generated. In this way I will be able to 
differentiate the DNS traffic per client and avoid accounting DNS 
traffic that the gateway generated for its own services.


It cannot be done because there is no 1:1 mapping between client and 
authoritative side of BIND. Multiple client queries might be solved by a 
single query to authoritative side, or a single query might cause 
multiple interrelated queries.


If money are involved then I say "don't even try": All reasonable 
solutions will cause either overcharging or undercharging, which is not 
only objectionable but also possibly illegal.


Out of curiosity, is the amount of traffic so large it is worth 
considering it? Compared to all the YouTube videos? :-)


--
Petr Špaček
--
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

DNS traffic tracking

[Alex K]

Hi all,

I have the following problem: I run a caching dns server using bind9
v9.10.3 in a gateway device which it serves several internal LAN IP
addresses (clients). I am doing some traffic accounting in the gateway
device using Linux conntrack so as to calculate the generated client
traffic (mostly HTTP/HTTPs related, in/out) so as to charge the volume
consumed.

What I cannot charge is the actual DNS traffic that each client is
generating, since each client DNS request is actually two sessions, one
between client and gateway device and the other between gateway and
upstream DNS servers. It seems to me not fare to charge the traffic
observed between the client and the gateway since the internal DNS traffic
includes cached responses and may be much higher from the actual DNS
traffic observed on the WAN side (gateway - upstream).

I was wondering if there is a solution to this. If bind9 has any feature
that can be used to track the WAN DNS traffic and understand from which
client was first requested/generated. In this way I will be able to
differentiate the DNS traffic per client and avoid accounting DNS traffic
that the gateway generated for its own services.

Just as an additional note on this, I had in the past the same issue with
the proxy traffic that this same gateway was generating and found a
solution by using TPROXY feature of the squid proxy, which exposes the real
internal client IP address at the WAN traffic which can later be NATed.

Thanx for any ideas,
Alex
-- 
Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from 
this list

ISC funds the development of this software with paid support subscriptions. 
Contact us at https://www.isc.org/contact/ for more information.


bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users