Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
On 12/29/2011 12:55 AM, Fabio Sangiovanni wrote: Hello list, I'm a newbie of pdns and I'm setting up a recursor-only installation as local dns cache for a high load smtp server. I'm using the lua scripting feature as a way to track down MX queries. In particular, the preresolve() function relies on luasql to execute queries on a (local) mysql database. This is my problem: if I enable more than one thread in recursor.conf I get the lua script loaded more than once, and ultimately concurrent queries to the db that could mess up the stored data, depending on the execution order of select and update queries by the various threads. I'd like to avoid circumventing these race conditions through complicated lua coding, and instead I'd prefer to keep things as simple as possible and limit the number of pdns threads to just 1 (threads=1 setting in recursor.conf). I know that the lua feature is not meant to be used this way, but this could really help me solving a big problem :) Since I'm just hosting a tiny little site I'm probably missing something - but can you please explain what you're trying to accomplish? If you're just logging MX queries as they occur - what's the race condition that you see occurring? Are you totaling requests by client IP - in which case I don't see a concern, or by the target address? If by the target address, could you not use a stored procedure that basically executes lock table ; update bigtable set queries = queries + 1 where mxip = parameter ; unlock table (I don't use mysql so I don't know mysql-specific syntax - nor how that would impact performance). Alternatively, have your lua script only add records to a temporary table - so there would be potentially multiple entries for a given target address. Then have a cron job periodically grab the temporary table, add the values to the primary totals table, and delete the temporary values. -- Daniel ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
On Jan 5, 2012, at 4:31 , Daniel L. Miller wrote: lock table ; update bigtable set queries = queries + 1 where mxip = parameter ; unlock table This update-query does not have a race-condition. The locks are not necessary :) Kind regards, Peter van Dijk ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
Hi Bert, are you planning to run multiple Lua sessions per thread in the future (in PowerDNS Recursor)? If yes, do you think it will be possible to have a switch for this feature in recursor.conf? I need to make a decision for my company about the use of PowerDNS Recursor in a project and everything depends on the possibility of having one single lua session per thread. Thanks a lot for your support. Cheers, Fabio Il giorno 30/dic/2011, alle ore 09:43, bert hubert ha scritto: On Thu, Dec 29, 2011 at 08:55:57AM +, Fabio Sangiovanni wrote: am I right thinking this would solve my problem? For now, and for this version of the PowerDNS Recursor, if you set threads to 1, you will really only get 1 simultaneous Lua session. However, this is not true in other parts of PowerDNS. In the authoritative server if you set 'distributor-threads=1' you might still get several parallel database connections. If you are going to do synchronous SQL updates from Lua, do realize that these typically cost up to a millisecond including latency. This means that your Resolver will peak at 1000 qps. how deep is the impact on performances changing the number of threads from 2 (default setting) to just 1? are there other sensible settings that I could adjust to better suit my needs? You might want to look at other ways of doing statistics if you truly want high performance. PowerDNS right now does not offer a good interface for very rapid query logging, but perhaps you could use a textfile from Lua. Good luck! Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
Hi Nils, thanks for your reply. Database transactions help in terms of integrity, but they can do nothing about the order of data updates. If 2 threads work independently and execute the same update query on the db, I can't predict with certainty which one will occur first. Fabio Il giorno 30/dic/2011, alle ore 00:32, Nils Breunese (Lemonbit) ha scritto: Fabio Sangiovanni wrote: I'm a newbie of pdns and I'm setting up a recursor-only installation as local dns cache for a high load smtp server. I'm using the lua scripting feature as a way to track down MX queries. In particular, the preresolve() function relies on luasql to execute queries on a (local) mysql database. This is my problem: if I enable more than one thread in recursor.conf I get the lua script loaded more than once, and ultimately concurrent queries to the db that could mess up the stored data, depending on the execution order of select and update queries by the various threads. I'd like to avoid circumventing these race conditions through complicated lua coding, and instead I'd prefer to keep things as simple as possible and limit the number of pdns threads to just 1 (threads=1 setting in recursor.conf). I know that the lua feature is not meant to be used this way, but this could really help me solving a big problem :) Now my questions are: am I right thinking this would solve my problem? how deep is the impact on performances changing the number of threads from 2 (default setting) to just 1? are there other sensible settings that I could adjust to better suit my needs? Wouldn't using database transactions solve your integrity problem? Nils. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
Fabio Sangiovanni wrote: Database transactions help in terms of integrity, but they can do nothing about the order of data updates. If 2 threads work independently and execute the same update query on the db, I can't predict with certainty which one will occur first. AFAIK no updates are committed until the transaction is committed, so all queries within a transaction are guaranteed to happen as if they were one modification to the database. If the order of transactions is important then I think you might need to rethink your logic, because then the whole thing would need to happen in a transaction. I'm not sure what you're trying to do exactly, but it sounds like transactions should be able to help here. Nils. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
Hi, thanks for your reply. Il giorno 30/dic/2011, alle ore 09:43, bert hubert ha scritto: On Thu, Dec 29, 2011 at 08:55:57AM +, Fabio Sangiovanni wrote: am I right thinking this would solve my problem? For now, and for this version of the PowerDNS Recursor, if you set threads to 1, you will really only get 1 simultaneous Lua session. Do you mean that this may change in the future? Will PowerDNS Recursor eventually run more Lua sessions per thread? However, this is not true in other parts of PowerDNS. In the authoritative server if you set 'distributor-threads=1' you might still get several parallel database connections. It's ok for me, since I'm using the PowerDNS Recursor standalone, without the autoritative server (not useful in my use case). If you are going to do synchronous SQL updates from Lua, do realize that these typically cost up to a millisecond including latency. This means that your Resolver will peak at 1000 qps. With just 1 Lua session running I can rework the Lua code: I can schedule the import of data from db every 5 minutes, and use ad in-memory data structure for the computation I need to be done. I can also ditch the synchronous SQL updates, that I used just to have consistency between threads (wrong assumption, though). The performance of course should increase this way, providing that just one PowerDNS thread is ok (the server is a high traffic smtp). This logic wouldn't work with more than one thread, since each would have its own data structure. Good luck! Bert Thanks! Fabio ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
Ok I'm kind of a newbie here :) My understanding of the problem is: if I have 2 threads, running at the same time, both of them executing the same SQL transaction (made up of more select or update queries), how can I predict which one is executed first? It's just like you said: I'd need the whole thing to happen in a single inter-thread transaction :) The solution, for now, is limiting Lua sessions to 1 lowering the total number of threads to 1. Fabio Il giorno 30/dic/2011, alle ore 10:32, Nils Breunese (Lemonbit) ha scritto: Fabio Sangiovanni wrote: Database transactions help in terms of integrity, but they can do nothing about the order of data updates. If 2 threads work independently and execute the same update query on the db, I can't predict with certainty which one will occur first. AFAIK no updates are committed until the transaction is committed, so all queries within a transaction are guaranteed to happen as if they were one modification to the database. If the order of transactions is important then I think you might need to rethink your logic, because then the whole thing would need to happen in a transaction. I'm not sure what you're trying to do exactly, but it sounds like transactions should be able to help here. Nils. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
On Fri, Dec 30, 2011 at 10:00:50AM +, Fabio Sangiovanni wrote: Ok I'm kind of a newbie here :) My understanding of the problem is: if I have 2 threads, running at the same time, both of them executing the same SQL transaction (made up of more select or update queries), how can I predict which one is executed first? It's just like you said: I'd need the whole thing to happen in a single inter-thread transaction :) The solution, for now, is limiting Lua sessions to 1 lowering the total number of threads to 1. Fabio Il giorno 30/dic/2011, alle ore 10:32, Nils Breunese (Lemonbit) ha scritto: Fabio Sangiovanni wrote: Database transactions help in terms of integrity, but they can do nothing about the order of data updates. If 2 threads work independently and execute the same update query on the db, I can't predict with certainty which one will occur first. AFAIK no updates are committed until the transaction is committed, so all queries within a transaction are guaranteed to happen as if they were one modification to the database. If the order of transactions is important then I think you might need to rethink your logic, because then the whole thing would need to happen in a transaction. I'm not sure what you're trying to do exactly, but it sounds like transactions should be able to help here. Nils. I can't help wondering if what you are doing is going to look like UPDATE stats SET counter = counter + 1 WHERE foo = 'bar' If so, you shouldn't have any issues with concurrency afaik. Running Lua SQL script in high traffic site however is probably not a very good idea due to aforementioned perfomance issues. Aki Tuomi signature.asc Description: Digital signature ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
[Pdns-users] pdns recursor - unthreaded vs threaded operation
Hello list, I'm a newbie of pdns and I'm setting up a recursor-only installation as local dns cache for a high load smtp server. I'm using the lua scripting feature as a way to track down MX queries. In particular, the preresolve() function relies on luasql to execute queries on a (local) mysql database. This is my problem: if I enable more than one thread in recursor.conf I get the lua script loaded more than once, and ultimately concurrent queries to the db that could mess up the stored data, depending on the execution order of select and update queries by the various threads. I'd like to avoid circumventing these race conditions through complicated lua coding, and instead I'd prefer to keep things as simple as possible and limit the number of pdns threads to just 1 (threads=1 setting in recursor.conf). I know that the lua feature is not meant to be used this way, but this could really help me solving a big problem :) Now my questions are: am I right thinking this would solve my problem? how deep is the impact on performances changing the number of threads from 2 (default setting) to just 1? are there other sensible settings that I could adjust to better suit my needs? Thanks a lot for your support. Cheers, Fabio ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] pdns recursor - unthreaded vs threaded operation
Fabio Sangiovanni wrote: I'm a newbie of pdns and I'm setting up a recursor-only installation as local dns cache for a high load smtp server. I'm using the lua scripting feature as a way to track down MX queries. In particular, the preresolve() function relies on luasql to execute queries on a (local) mysql database. This is my problem: if I enable more than one thread in recursor.conf I get the lua script loaded more than once, and ultimately concurrent queries to the db that could mess up the stored data, depending on the execution order of select and update queries by the various threads. I'd like to avoid circumventing these race conditions through complicated lua coding, and instead I'd prefer to keep things as simple as possible and limit the number of pdns threads to just 1 (threads=1 setting in recursor.conf). I know that the lua feature is not meant to be used this way, but this could really help me solving a big problem :) Now my questions are: am I right thinking this would solve my problem? how deep is the impact on performances changing the number of threads from 2 (default setting) to just 1? are there other sensible settings that I could adjust to better suit my needs? Wouldn't using database transactions solve your integrity problem? Nils. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
