Paul G. Allen wrote: > On Tue, 2007-03-13 at 16:43 -0800, Paul G. Allen wrote: > >> Turns out we are black listed. The IP that is listed is 216.70.148.112. >> That's not our mail server either. In the past day I've found several >> things contributing to this problem. Once resolved I'll try to remember >> to post what I can here so others may learn from our mistakes (and learn >> why your IP(s) may get listed even though they shouldn't). >> > > OK, here's the end of the story. > > There were two causes of the black listing, and two causes of our IP > being re-added to the lists after having been removed. The causes of the > black listing can come later. The re-listing is the important part of > showing where a mis-configured network can bite you in the butt. Kids, > don't do this at home (or at your company). Most of you on this list > probably do things the right way, but those who don't may learn from > this. BTW, I only became involved in this because our IT guy is on a > trip this week and the proverbial crap only hit the fan right after he > left. I am filling in for him this week. > > > Problem 1. > > Our mail server (Exchange) has an external IP address, different from > our firewall. It is NAT'd through the firewall, and this should be a > Good Thing. However, when the mail server sends email, the outgoing IP > is not NAT'd to its IP, and instead uses the firewall IP. This is > *wrong* *wrong* *wrong*. Why? > > When a slew of e-mails were sent from a user's computer (can you say > trojan?!), they were sent using the trojan's SMTP server and sent > through the firewall *using the firewall IP*. The result is that instead > of our firewall IP being listed, with no ill effect to the mail server, > essentially the firewall *and* mail server were listed. > > When mail is sent through the one and only legitimate corporate mail > server, it should use its own IP address. That way if any other machine > on the network does a Bad Thing, the Bad Thing does not appear to be > coming from the mail server. > > > Problem 2. > > Not filtering or blocking port 25 on the firewall so that nothing other > than those servers allowed to send mail, can send e-mail. Need I say > more? > > > Problem 3. > > The PTR records and MX records on our ISP DNS servers are not configured > to support reverse DNS. Usually not a big deal, but on occasion there's > a system out there that wants to do a reverse lookup to see if the > computer sending the e-mail (or otherwise requesting a connection) is > who it says it is. In such a case, if the e-mail Received: portion of > the header does not match the reverse DNS, then the mail is rejected. > Reverse DNS in some cases must be requested of the ISP or they won't > provide it. > > > Problem 4. > > Our mail server's HELO response was "quakeglobal.com" when it should > have been "mail.quakeglobal.com". This posed a big problem when trying > to de-list our IP. The CBL list wants to see a FQDN (Fully Qualified > Domain Name) as the name given in the HELO response from the mail > server. If it does not, it will re-list the IP. Because several other > lists use CBL as a reference, our IP remained listed on all of them > until the mail server response was corrected. Because several spam > filters use different lists, many people/companies could not receive our > e-mails. > > > Problem 5. > > Our ISP did not update the ARIN records to indicate that our IP block > was ours. It was listed as belonging to a previous customer of the ISP. > So, doing a whois would come up with conflicting results. Even though > the ISP information was correct in all cases, going directly to ARIN > gave the wrong organization and contact address. Looking just now the > record still has not been updated, but they told us they would update it > ASAP. > > > The source of the listings. > > Now, for how we got listed to begin with. Two reasons. First another > server other than our mail server sent a bunch of e-mails as part of an > engineering test. It sent too many too fast (engineering mistake). As > outlined above, these e-mails had the same source address as our > corporate mail server. When it was black listed, so was our corporate > mail. > > The second problem was a trojan that infected a new computer. The trojan > was sending spam about buying pills online. Since port 25 was not > filtered, and the IP was again the same as the mail server,... well you > get it. > > > The Summary. > > So, some very basic oversights and mistakes caused several days of > corporate wide problems. Some things that aren't usually a big deal > (like reverse DNS, a HELO response that's not a FQDN) only need to rear > their head once to cause you as an admin and your company as a whole a > ton of trouble. > > I won't even get in to the anti-spyware/virus issues and the use of > Outhouse. >
Great summary! Thanks for posting. I presume the text will be carved on stone wall of the sysadmin's cell in your IT monastery (or wherever else your IT policies are documented). Regards, ..jim -- [email protected] http://www.kernel-panic.org/cgi-bin/mailman/listinfo/kplug-list
