On Tue, 2007-03-13 at 16:43 -0800, Paul G. Allen wrote: > > > > Turns out we are black listed. The IP that is listed is 216.70.148.112. > That's not our mail server either. In the past day I've found several > things contributing to this problem. Once resolved I'll try to remember > to post what I can here so others may learn from our mistakes (and learn > why your IP(s) may get listed even though they shouldn't). >
OK, here's the end of the story. There were two causes of the black listing, and two causes of our IP being re-added to the lists after having been removed. The causes of the black listing can come later. The re-listing is the important part of showing where a mis-configured network can bite you in the butt. Kids, don't do this at home (or at your company). Most of you on this list probably do things the right way, but those who don't may learn from this. BTW, I only became involved in this because our IT guy is on a trip this week and the proverbial crap only hit the fan right after he left. I am filling in for him this week. Problem 1. Our mail server (Exchange) has an external IP address, different from our firewall. It is NAT'd through the firewall, and this should be a Good Thing. However, when the mail server sends email, the outgoing IP is not NAT'd to its IP, and instead uses the firewall IP. This is *wrong* *wrong* *wrong*. Why? When a slew of e-mails were sent from a user's computer (can you say trojan?!), they were sent using the trojan's SMTP server and sent through the firewall *using the firewall IP*. The result is that instead of our firewall IP being listed, with no ill effect to the mail server, essentially the firewall *and* mail server were listed. When mail is sent through the one and only legitimate corporate mail server, it should use its own IP address. That way if any other machine on the network does a Bad Thing, the Bad Thing does not appear to be coming from the mail server. Problem 2. Not filtering or blocking port 25 on the firewall so that nothing other than those servers allowed to send mail, can send e-mail. Need I say more? Problem 3. The PTR records and MX records on our ISP DNS servers are not configured to support reverse DNS. Usually not a big deal, but on occasion there's a system out there that wants to do a reverse lookup to see if the computer sending the e-mail (or otherwise requesting a connection) is who it says it is. In such a case, if the e-mail Received: portion of the header does not match the reverse DNS, then the mail is rejected. Reverse DNS in some cases must be requested of the ISP or they won't provide it. Problem 4. Our mail server's HELO response was "quakeglobal.com" when it should have been "mail.quakeglobal.com". This posed a big problem when trying to de-list our IP. The CBL list wants to see a FQDN (Fully Qualified Domain Name) as the name given in the HELO response from the mail server. If it does not, it will re-list the IP. Because several other lists use CBL as a reference, our IP remained listed on all of them until the mail server response was corrected. Because several spam filters use different lists, many people/companies could not receive our e-mails. Problem 5. Our ISP did not update the ARIN records to indicate that our IP block was ours. It was listed as belonging to a previous customer of the ISP. So, doing a whois would come up with conflicting results. Even though the ISP information was correct in all cases, going directly to ARIN gave the wrong organization and contact address. Looking just now the record still has not been updated, but they told us they would update it ASAP. The source of the listings. Now, for how we got listed to begin with. Two reasons. First another server other than our mail server sent a bunch of e-mails as part of an engineering test. It sent too many too fast (engineering mistake). As outlined above, these e-mails had the same source address as our corporate mail server. When it was black listed, so was our corporate mail. The second problem was a trojan that infected a new computer. The trojan was sending spam about buying pills online. Since port 25 was not filtered, and the IP was again the same as the mail server,... well you get it. The Summary. So, some very basic oversights and mistakes caused several days of corporate wide problems. Some things that aren't usually a big deal (like reverse DNS, a HELO response that's not a FQDN) only need to rear their head once to cause you as an admin and your company as a whole a ton of trouble. I won't even get in to the anti-spyware/virus issues and the use of Outhouse. PGA -- [email protected] http://www.kernel-panic.org/cgi-bin/mailman/listinfo/kplug-list
