Re: [Pdns-users] Dnstap and kafka
OK. Thanks for clarification. On Thu, Jan 14, 2021, 4:00 PM Brian Candler wrote: > On 14/01/2021 13:11, Michael Chisina wrote: > > Thanks for the info > > > > # is there any need for middleware software if I use dtap (dnstap > > favoured) following from github > > > > https://github.com/mimuret/dtap#kafka > > <https://github.com/mimuret/dtap#kafka> > > > That *is* the middleware software. > > ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Dnstap and kafka
Thanks for the info # is there any need for middleware software if I use dtap (dnstap favoured) following from github https://github.com/mimuret/dtap#kafka And also configure the following: Kafka Make flatting DNSTAP message,And it forawrd to kafka host. [[OutputKafks]] Hosts = ["kafka.example.jp:9092"] Topic = "dnstap_message" # the dataflow will be as follows: DNS message --->dnstap--->middleware--->Kafka producer connector ---> Kafka streaming---> Kafka consumer connector---> debezium connector --->timescaledb(postgresql favoured) Regards Michael Chisina On Wed, Jan 13, 2021, 5:27 PM Brian Candler wrote: > On 13/01/2021 14:58, Michael Chisina via Pdns-users wrote: > > I want to frame stream powerdns recursor DNS query and response using > dnstap to an apache kafka remote server (202.20.20.1). > # what are the configurations needed on recursor? > > dnstap doesn't talk to kafka. > > You'll need to run some middleware software which accepts dnstap > <https://dnstap.info/>-formatted messages, and writes them to kafka - > which could be in raw dnstap form, or decoded into some other form (e.g. > JSON). Google "dnstap kafka" for some options, or write your own. > > Then you configure pdns-recursor to send dnstap messages to this > middleware server. > > If you run the middleware on the same server as pdns-recursor, then they > can communicate over a unix domain socket. If you want to communicate to a > remote server over TCP, then you will need a sufficiently new version of > libfstrm which supports this. In practice this means that if you are using > Ubuntu then you need 20.04 not 18.04. See this thread: > > https://mailman.powerdns.com/pipermail/pdns-users/2020-June/026724.html > > > # what is dns message schema(s) format for the database creation? > > kafka isn't a database. If you want to write these messages to a > database, then that's an additional step. You'll need some more software > which reads messages from kafka, decodes them, and writes to a database in > some schema that you define. Beware that a busy recursor can generate a > very large volume of messages, so you might want to aggregate them first. > > HTH, > > Brian. > ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
[Pdns-users] Dnstap and kafka
: Hello, I want to frame stream powerdns recursor DNS query and response using dnstap to an apache kafka remote server (202.20.20.1). # what are the configurations needed on recursor? # what is dns message schema(s) format for the database creation? Regards Michael Chisina ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Contents of Pdns-users digest
I think load balancer is the best option and configure the policy(ies) on it. Michael Chisina On Wed, Nov 13, 2019, 2:00 PM wrote: > Send Pdns-users mailing list submissions to > pdns-users@mailman.powerdns.com > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.powerdns.com/mailman/listinfo/pdns-users > or, via email, send a message with subject or body 'help' to > pdns-users-requ...@mailman.powerdns.com > > You can reach the person managing the list at > pdns-users-ow...@mailman.powerdns.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Pdns-users digest..." > > > Today's Topics: > >1. Re: Forward client request (Thomas Mieslinger) > > > -- > > Message: 1 > Date: Tue, 12 Nov 2019 15:37:32 +0100 > From: Thomas Mieslinger > To: pdns-users@mailman.powerdns.com > Subject: Re: [Pdns-users] Forward client request > Message-ID: <78ad5583-6f4e-a150-86c0-eece65eee...@mail.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > If you are familiar with policy based routing or multiple vrfs and > running multiple pdns_recursor instances you could create two (or more) > pdns_recursor services and configure the routing to send the dns > requests to the desired recursor. > > On 11/12/19 12:05 PM, mendisobal via Pdns-users wrote: > > How i can forward recursive DNS requests based on source address of the > > client. To do this need to have ability to return address of the next NS > > from preresove function (instead of NS-records). > > Is there any example on lua? > > > > ___ > > Pdns-users mailing list > > Pdns-users@mailman.powerdns.com > > https://mailman.powerdns.com/mailman/listinfo/pdns-users > > > > > -- > > Subject: Digest Footer > > ___ > Pdns-users mailing list > Pdns-users@mailman.powerdns.com > https://mailman.powerdns.com/mailman/listinfo/pdns-users > > > -- > > End of Pdns-users Digest, Vol 202, Issue 7 > ** > ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
