[vchkpw] Re: block non-relay from remote to local?
Hello Billy, On Saturday, July 2, 2005 at 6:32:47 PM Billy wrote: N.B.: Number of authentication should not play a role in accessing your cdb-file, if you're configured vpopmail to only use MySQL the cdb will be as static as your kernel: unless *you* change it, it won't change. (I'm going by memory, so this is a paraphrase.) If you aren't familiar with the Matt Simerson mysql patch, it was born because there can be major lookup problems with the cdb file, especially using POP before SMTP. Have tested it a long time ago and know why it was developed, albeit I don't actively use it. Mysql has no problem with the above scenario, as it is designed for heavy accesses and changes to its tables. Absolutely right. But: what's the matter with 'POP-before-SMTP done through MySQL' and additionally using a .cdb-file for static entries? Does the patch nevertheless a MySQL-lookup, even if something is found in .cdb-file? In this case a .cdb-file in fact wouldn't make much sense, except the fact the answer from MySQL could kept short (no result) and some parsing time could be spared. -- Best regards Peter Palmreuther An authority: someone who knows lots of things you could care less about.
Re: [vchkpw] Re: block non-relay from remote to local?
Peter Palmreuther wrote: Hello Billy, On Saturday, July 2, 2005 at 6:32:47 PM Billy wrote: N.B.: Number of authentication should not play a role in accessing your cdb-file, if you're configured vpopmail to only use MySQL the cdb will be as static as your kernel: unless *you* change it, it won't change. (I'm going by memory, so this is a paraphrase.) If you aren't familiar with the Matt Simerson mysql patch, it was born because there can be major lookup problems with the cdb file, especially using POP before SMTP. Have tested it a long time ago and know why it was developed, albeit I don't actively use it. Mysql has no problem with the above scenario, as it is designed for heavy accesses and changes to its tables. Absolutely right. But: what's the matter with 'POP-before-SMTP done through MySQL' and additionally using a .cdb-file for static entries? Does the patch nevertheless a MySQL-lookup, even if something is found in .cdb-file? In this case a .cdb-file in fact wouldn't make much sense, except the fact the answer from MySQL could kept short (no result) and some parsing time could be spared. Well, I had to look up the stuff myself. I'm not completely positive, but it looks like you can still use -x and the (Matt Simerson hack) -S, too. See http://www.tnpi.biz/internet/mail/toaster/patches/tcpserver-mysql.shtml I use a lot of Matt's stuff, but as you can see, this hack is for the big leagues. (I just looked at my service run files, which are automatically generated. No -S, just -x /usr/local/vpopmail/etc/tcp.smtp.cdb -- I don't do pop-before-smtp) But using both, you do two reads per SMTP access. Only consider that if the cdb file is static, that small of a file will be resident in your memory cache almost assuradly. I think you are splitting hairs, until you get a 10,000+ user system and benchmark it. You will need to see the source hack to see what is done, when, and how, or benchmark it. Or ask Matt! Billy
Re: [vchkpw] Re: block non-relay from remote to local?
Peter Palmreuther wrote: YMMD, but 'fopen()', a fast, hash-driven, seek (the way 'cdb' works) and a quick 'read' for a few bytes should be less overhead than a complete SQL query, including parsing the result. Even if your MySQL would run locally and accessed through UNIX-socket I'd expect it to be not only more overhead, but also taking more time than this quick local, read-only, precise access in a small file. But it's your system, whatever makes you happy and serves your needs can be your solution. N.B.: Number of authentication should not play a role in accessing your cdb-file, if you're configured vpopmail to only use MySQL the cdb will be as static as your kernel: unless *you* change it, it won't change. (I'm going by memory, so this is a paraphrase.) If you aren't familiar with the Matt Simerson mysql patch, it was born because there can be major lookup problems with the cdb file, especially using POP before SMTP. Imagine the POP server populating the text file (and re-compiling the CDB) 10 times per second. Now, imagine 20 queries per second on the CDB file, that in some cases is in the middle of a file alteration. The disk, in cases like this, hardly ever gets to writing the file from a kernel buffer, so what you are seeing is memory accesses on this file most of the time. There will simply be cases where the CDB file gets hammered too hard and corrupted. It will need attention by the admin during peak hours. Mysql has no problem with the above scenario, as it is designed for heavy accesses and changes to its tables. By the way, a way to solve the original problem sound to me the JOB FOR a FIREWALL and ROUTER! I am not sure if the server in question has one or two Ethernet interfaces, but if it doesn't, they often cost about $10 to $30 (unless they both need global IP's). If you route inbound mail from your upstream MXs to one interface (say, fxp0) and that is the only source of port 25 traffic from the global internet, you could have qmail listen to that interface. Firewall setup is simple -- only allow the MX servers to talk to that port 25. Meanwhile, the internal port 25 traffic (which as another topic should be port 587) can come into the other interface, say fxp1. The firewall would need no restriction for this interface. As my other clue, your customer and others should get used to using port 587 as their SMTP relay port, rather than port 25. That way, some of your customer's users could be on the global Internet, and still send mail to their firewalled-port-25-is-illegal mail server all day on the submission port 587. It would work internally, too. So here is a summary: fxp0 - global internet -- inbound port 25 only allowed from 3 IP addresses. port 587 is allowed for SMTP AUTH. fxp1 - internal net like 10.0.0.21 -- inbound port 25 and port 587 is allowed for SMTP or SMTP AUTH. Billy
Re: [vchkpw] Re: block non-relay from remote to local?
On Jul 2, 2005, at 9:32 AM, Billy Newsom wrote: If you aren't familiar with the Matt Simerson mysql patch, it was born because there can be major lookup problems with the cdb file, especially using POP before SMTP. Imagine the POP server populating the text file (and re-compiling the CDB) 10 times per second. Now, imagine 20 queries per second on the CDB file, that in some cases is in the middle of a file alteration. The disk, in cases like this, hardly ever gets to writing the file from a kernel buffer, so what you are seeing is memory accesses on this file most of the time. There will simply be cases where the CDB file gets hammered too hard and corrupted. It will need attention by the admin during peak hours. If I recall the thread though, the server in question is NOT set up for POP before SMTP. So, the cdb file is static and does not get updated. If it has 5 or so entries for the servers that will feed it, I would have to agree that a cdb lookup would to a static file of 5 entries would be more efficient than a MySQL query (which has to be parsed by the MySQL server) for the same lookup. -- Tom Collins - [EMAIL PROTECTED] QmailAdmin: http://qmailadmin.sf.net/ Vpopmail: http://vpopmail.sf.net/ You don't need a laptop to troubleshoot high-speed Internet: sniffter.com
Re: [vchkpw] Re: block non-relay from remote to local?
At 09:32 AM 7/2/2005, you wrote: Mysql has no problem with the above scenario, as it is designed for heavy accesses and changes to its tables. yup. my mysql server (which is on the vpopmail server, of course, so also handling httpd, imap, pop3 as well), which handles vpopmail, horde/IMP, twig, squirrel calendaring, and cacti, averages - long term - about 20qps. during peak parts of the day, the average is 40qps, with spikes to 150qps or more. never breaks a sweat, and this is on hardware that's pathetic by current standards (dual ultrasparc II 300Mhz). By the way, a way to solve the original problem sound to me the JOB FOR a FIREWALL and ROUTER! I am not sure if the server in question has one or two Ethernet interfaces, but if it doesn't, they often cost about $10 to $30 (unless they both need global IP's). If you route inbound mail from your upstream MXs to one interface (say, fxp0) and that is the only source of port 25 traffic from the global internet, you could have qmail listen to that interface. Firewall setup is simple -- only allow the MX servers to talk to that port 25. My MX servers only accept connections on port 25, and port 32 (otherwise known as port 22 - but i got sick of the stupid script-kiddies hammering my port 22 all day long, so i moved it to 32). I do also run a restricted FTP server on the MX servers for remote probing and statistics. Meanwhile, the internal port 25 traffic (which as another topic should be port 587) can come into the other interface, say fxp1. The firewall would need no restriction for this interface. there is no 'internal' port 25 traffic. My service provides email service for businesses. I'm not an ISP. all traffic to my servers is inbound from the global internet. note also that there is *no* reason for anyone to use port 587. more below. As my other clue, your customer and others should get used to using port 587 as their SMTP relay port, rather than port 25. That way, some of your customer's users could be on the global Internet, and still send mail to their firewalled-port-25-is-illegal mail server all day on the submission port 587. It would work internally, too. We provide alternate access to our SMTP server for those customer's whose ISP's block port 25. We use port 2525. what, you say? 2525 is registered to MS V Worlds. my response is, so freaking what? *there are no restrictions on the use of registered ports for any service one desires*. true, i haven't spent a lot of time checking the RFC's. but i'm pretty sure that IANA's 'rules' are only 'recommendations'. 587 is dandy, but it's also another random string of digits for customers to try to remember. 2525 is easy for customers to remember. if it should ever conflict with someone's use of MS V Worlds, well by gosh we'll just start another server on another port just for them. I'm not holding my breath. So here is a summary: fxp0 - global internet -- inbound port 25 only allowed from 3 IP addresses. port 587 is allowed for SMTP AUTH. fxp1 - internal net like 10.0.0.21 -- inbound port 25 and port 587 is allowed for SMTP or SMTP AUTH. it's a nice design, but not applicable to my setup. i suppose i could require *all* of my customers to start using port 2525. what a headache. ask hundreds of business to have their thousands of employees change their email client settings. oy vey. Paul Theodoropoulos http://www.anastrophe.com http://www.smileglobal.com
Re: [vchkpw] Re: block non-relay from remote to local?
At 09:54 AM 7/2/2005, Tom Collins wrote: If I recall the thread though, the server in question is NOT set up for POP before SMTP. So, the cdb file is static and does not get updated. If it has 5 or so entries for the servers that will feed it, I would have to agree that a cdb lookup would to a static file of 5 entries would be more efficient than a MySQL query (which has to be parsed by the MySQL server) for the same lookup. nope, in this case, i am indeed using pop before smtp. as noted in my last message, my mysql server averages 20qps, runs at 40qps during the busy part of the day, with peaks in excess of 150qps, and never breaks a sweat. i last restarted my mysql server on june 23 - since then, there have been 4 Slow_queries. Paul Theodoropoulos http://www.anastrophe.com http://www.smileglobal.com
Re: [vchkpw] Re: block non-relay from remote to local?
Paul Theodoropoulos wrote: At 09:32 AM 7/2/2005, you wrote: there is no 'internal' port 25 traffic. My service provides email service for businesses. I'm not an ISP. all traffic to my servers is inbound from the global internet. I guess I was looking at your customer as having an SMTP relay server at his site. That's just the way I read your original post. I was talking about the customer's system needing the firewall, not yours -- I see now you were talking about your own POP server, not theirs. note also that there is *no* reason for anyone to use port 587. more below. As my other clue, your customer and others should get used to using port 587 as their SMTP relay port, rather than port 25. That way, some of your customer's users could be on the global Internet, and still send mail to their firewalled-port-25-is-illegal mail server all day on the submission port 587. It would work internally, too. We provide alternate access to our SMTP server for those customer's whose ISP's block port 25. We use port 2525. what, you say? 2525 is registered to MS V Worlds. my response is, so freaking what? *there are no restrictions on the use of registered ports for any service one desires*. true, i haven't spent a lot of time checking the RFC's. but i'm pretty sure that IANA's 'rules' are only 'recommendations'. 587 is dandy, but it's also another random string of digits for customers to try to remember. 2525 is easy for customers to remember. if it should ever conflict with someone's use of MS V Worlds, well by gosh we'll just start another server on another port just for them. I'm not holding my breath. Well, I can't say I didn't do the the same thing until recently. I chose my own secret port number to bypass a port 25 block. Blocking port 25 is becoming a major reality now. I was merely saying that there is a standard way to allow things to happen. You will see back there at Matt Simerson's site that he is now getting qmail to effectively listen on SMTP and submission ports to start abinding by the RFC for roaming users. Since it is a rather new phenonimon, not many know about it, but as more ISP's block and more mail providers (like you and I) try to avoid these issues, the port 587 number will become fairly well-known. And, by the way, in the case of a clueless user anyway, one port number is just as hard as another to use, as they will need a lot of handholding to setup their client. And for those who get the idea, port 587 will eventually be memorable. We weren't around when the RFC got written, or we might have tried for a smarter port number. In any case, I only feel that once I catch a clue, I might as well start using the right port number. I just opened up both the one I picked and 587, and determined to stick with the published standard unless necessary. Rumors persist that some ISP's might block port 587, but that is mostly hearsay. Billy
Re: [vchkpw] Re: block non-relay from remote to local?
At 10:59 AM 7/2/2005, you wrote: Well, I can't say I didn't do the the same thing until recently. I chose my own secret port number to bypass a port 25 block. Blocking port 25 is becoming a major reality now. I was merely saying that there is a standard way to allow things to happen. You will see back there at Matt Simerson's site that he is now getting qmail to effectively listen on SMTP and submission ports to start abinding by the RFC for roaming users. Since it is a rather new phenonimon, not many know about it, but as more ISP's block and more mail providers (like you and I) try to avoid these issues, the port 587 number will become fairly well-known. And, by the way, in the case of a clueless user anyway, one port number is just as hard as another to use, as they will need a lot of handholding to setup their client. And for those who get the idea, port 587 will eventually be memorable. We weren't around when the RFC got written, or we might have tried for a smarter port number. In any case, I only feel that once I catch a clue, I might as well start using the right port number. I just opened up both the one I picked and 587, and determined to stick with the published standard unless necessary. Rumors persist that some ISP's might block port 587, but that is mostly hearsay. yep, all good points. it certainly won't *hurt* anything for me to fire up another smtp on the submission port. as it is, the ports for mail i see codified are: 25 SMTP 465 SMTP-SSL (but shared with cisco's URD) 587 Submission 110 POP3 995 POP3-TLS/SSL 143 IMAP4 993 IMAP-TLS/SSL eek. i suppose for the average user, they're *all* random numbers, really. Paul Theodoropoulos http://www.anastrophe.com http://www.smileglobal.com
[vchkpw] Re: block non-relay from remote to local?
Hello Paul, On Friday, July 1, 2005 at 1:05:50 AM Paul wrote: okay, i presume then that there's a way to add permanent relay entries into the mysql database. no need for this, just edit your tcp.smtp file and add a static entry. i wish - except for this config option when i built vpopmail: --disable-rebuild-tcpserver-file \ neither my /service/smtpd/tcp.cdb nor /u1/etc/tcp.smtp.cdb are being accessed. vpopmail does not need to access these files. tcpserver needs to. Just add some static entries and tell tcpserver to use 'tcp.smtp.cdb' (which ever of the two you prefer) and let tcpserver *additionally* use MySQL database to look for dynamic relay allowances. -- Best regards Peter Palmreuther Wonder what my life would be without me.
Re: [vchkpw] Re: block non-relay from remote to local?
At 11:47 PM 6/30/2005, you wrote: i wish - except for this config option when i built vpopmail: --disable-rebuild-tcpserver-file \ neither my /service/smtpd/tcp.cdb nor /u1/etc/tcp.smtp.cdb are being accessed. vpopmail does not need to access these files. tcpserver needs to. Just add some static entries and tell tcpserver to use 'tcp.smtp.cdb' (which ever of the two you prefer) and let tcpserver *additionally* use MySQL database to look for dynamic relay allowances. sure - i know i can do that. but with the rate of connections and authentications that take place on my systems, i don't want to go back to the overhead of having that file touched multiple times per second. it was great when i recently upgraded to 5.4.10, as my old version didn't have that option. i've added some test entries to the relay db, dated ten years from now, which will likely do the trick of keeping them active. Paul Theodoropoulos http://www.anastrophe.com http://www.smileglobal.com
Re: [vchkpw] Re: block non-relay from remote to local?
On Friday 01 July 2005 02:09 am, Paul Theodoropoulos wrote: At 11:47 PM 6/30/2005, you wrote: i wish - except for this config option when i built vpopmail: --disable-rebuild-tcpserver-file \ neither my /service/smtpd/tcp.cdb nor /u1/etc/tcp.smtp.cdb are being accessed. vpopmail does not need to access these files. tcpserver needs to. Just add some static entries and tell tcpserver to use 'tcp.smtp.cdb' (which ever of the two you prefer) and let tcpserver *additionally* use MySQL database to look for dynamic relay allowances. sure - i know i can do that. but with the rate of connections and authentications that take place on my systems, i don't want to go back to the overhead of having that file touched multiple times per second. it was great when i recently upgraded to 5.4.10, as my old version didn't have that option. I think performing a mysql query on every connection is a bit more overhead than a CDB lookup ;) -Jeremy -- Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc. [EMAIL PROTECTED] ++ inter7.com ++ 866.528.3530 ++ 815.776.9465 int'l kitchen @ #qmail #gentoo on EFnet IRC ++ scriptkitchen.com/qmail GnuPG Key ID: 481BF7E2 ++ qmail wiki @ qmailwiki.org !! pgpPZa1PwgOLv.pgp Description: PGP signature
[vchkpw] Re: block non-relay from remote to local?
Hello Paul, On Friday, July 1, 2005 at 9:09:22 AM Paul wrote: i wish - except for this config option when i built vpopmail: --disable-rebuild-tcpserver-file \ neither my /service/smtpd/tcp.cdb nor /u1/etc/tcp.smtp.cdb are being accessed. vpopmail does not need to access these files. tcpserver needs to. Just add some static entries and tell tcpserver to use 'tcp.smtp.cdb' (which ever of the two you prefer) and let tcpserver *additionally* use MySQL database to look for dynamic relay allowances. sure - i know i can do that. but with the rate of connections and authentications that take place on my systems, i don't want to go back to the overhead of having that file touched multiple times per second. YMMD, but 'fopen()', a fast, hash-driven, seek (the way 'cdb' works) and a quick 'read' for a few bytes should be less overhead than a complete SQL query, including parsing the result. Even if your MySQL would run locally and accessed through UNIX-socket I'd expect it to be not only more overhead, but also taking more time than this quick local, read-only, precise access in a small file. But it's your system, whatever makes you happy and serves your needs can be your solution. N.B.: Number of authentication should not play a role in accessing your cdb-file, if you're configured vpopmail to only use MySQL the cdb will be as static as your kernel: unless *you* change it, it won't change. -- Best regards Peter Palmreuther Are you still here? The message is over. Go away!
