Re: Sender authentication zombies (was Re: Time to check the rate limits on your mail servers)
On 02/05/05, Douglas Otis [EMAIL PROTECTED] wrote: On Sat, 2005-02-05 at 19:10, J.D. Falk wrote: On 02/05/05, Douglas Otis [EMAIL PROTECTED] wrote: DK or IIM makes it clear who is administering the server and this authentication permits reputation assessment. Add an account identifier, and the problem is nailed. Ah, so you're saying that only the reputation of individual e-mail addresses is worth paying attention to? How do you expect that to scale to billions of messages per day? Without authenticating an identity, it must not be used in a reputation assessment. Currently this is commonly done by using the remote IP address authenticated through the action of transport. In the name space there are two options, the HELO and a validated signature. DK and IIM are attempting to allow the signature solution to scale. Heh, you don't need to convince me that DomainKeys is a good idea. I just don't see how you're jumping from the issue of end-user authentication (which is not free from zombies, as others have explained already) to domain-level reputation. Where's the link? If you're talking about adding user-level signatures to something like DomainKeys (which we already have in s/mime), how do you propose to scale that to interact with the reputation determination for billions of messages per day? -- J.D. Falk uncertainty is only a virtue [EMAIL PROTECTED]when you don't know the answer yet
Re: Sender authentication zombies (was Re: Time to check the rate limits on your mail servers)
On Sun, 2005-02-06 at 09:41, J.D. Falk wrote: On 02/05/05, Douglas Otis [EMAIL PROTECTED] wrote: Without authenticating an identity, it must not be used in a reputation assessment. Currently this is commonly done by using the remote IP address authenticated through the action of transport. In the name space there are two options, the HELO and a validated signature. DK and IIM are attempting to allow the signature solution to scale. Heh, you don't need to convince me that DomainKeys is a good idea. I just don't see how you're jumping from the issue of end-user authentication (which is not free from zombies, as others have explained already) to domain-level reputation. Where's the link? If you're talking about adding user-level signatures to something like DomainKeys (which we already have in s/mime), how do you propose to scale that to interact with the reputation determination for billions of messages per day? Take something like the DomainKey, and add an opaque identifier to the signature, derived from a user authentication process. This could be from an access server or a verification that resolves a dynamic address assignment to a persistent identifier. DomainKeys can scale. Adding an optional opaque persistent identifier will also scale, as this information is already collected. The reason for adding this identifier is two fold. One, it can be used to guard against replay attacks. Two, to assist in identifying compromised systems. Blocking by the provider scales; the identifier makes it easier to locate the problem via abuse reports. The signature ensurers that the provider can accurately attribute abuse. In addition, the provider should want to include the identifiers to protect their reputation in the face of a replay attack, which they can not block otherwise. By convention, the provider can publish their own identifier blackhole-listing just to address the replay attack, whereas known compromised systems should be blocked outright. The signature protects the provider from possible blocking and blackhole-listing errors, as the users will not believe they were the cause of their own problem. -Doug
Re: Time to check the rate limits on your mail servers
You should know all your users email addresses. You have got to be kidding. Not kidding. I have a mail system that handles mail for the example.com domain. I use SMTP AUTH as the only means to relay through the server. My expectation from my customers is that they will utilize this mail service for their [EMAIL PROTECTED] communications. This means the mail server has knowledge of all 'mail from' addresses my users are allowed to use. Who says that Joe ISP has to provide an open SMTP relay to all customers on his IP space? Let's face it, it doesn't work! Even with throttling some SPAM will make it thorough and tha mail server will be black listed and unable to deliver mail to many destinations in no time. It's only a matter of time before owned PCs aquire the 'intelligence' to utilize SMTP AUTH to relay mail. So to clarify my position, my SMTP server handles mail for my users and noone else. My users are identified by their email address(es) on my mail server. Therefore, I can enforce that may mailserver reject relayed mail that does not have a 'mail from' address that corresponds to one of the valid email addresses for an authenticated users. I am addressing the dilemma with the average home user. If you own a bunch of domains you're in a whole different class. Make arrangement with your ISP to handle your mail, run your own mail server or buy hosting with email accounts. Point is, if you own a bunch of domains you're not the average home user that floods the world with crap without their knowledge. Adi
Sender authentication zombies (was Re: Time to check the rate limits on your mail servers)
On 02/04/05, Douglas Otis [EMAIL PROTECTED] wrote: Attempting to detect spam trickled through thousands of compromised systems sent through the ISP's mail servers, SPF does nothing, Nor is it purported to. Domain-based authentication schemes are intended to handle an entirely different problem. and could actually damage the reputation of those domains that authorize the provider for their mailbox domain using SPF. These records can be read by the spammers and then exploited. Repairing this reputation could be next to impossible. You touch on some basic realities here: 1. spam coming out of your network will affect your reputation. 2. spam coming out of your own mail machines will affect your reputation even more immediately. Neither are affected by any of the domain authentication schemes currently in play (SPF, SenderID, DomainKeys, etc.) The spam itself may include forgeries, but that's a different issue. -- J.D. Falk uncertainty is only a virtue [EMAIL PROTECTED]when you don't know the answer yet
Re: Time to check the rate limits on your mail servers
TV Date: Fri, 4 Feb 2005 09:53:07 -0500 (EST) TV From: Todd Vierling TV The only way to be sure is via cryptographic signature. Barring that level False. You imply that a crypto signature is a perfect guarantee, and that nothing else can provide equal assurance. TV of immediate traceability, SPF provides a very useful data point to that TV end (as its *only* purpose is curbing forgery). SPF says mail from this domain should only come from these MXes. It doesn't stop someone from forging a random @domain.tld address from an SPF-blessed Everquick MX. Now, let's say it's known that Everquick MXes authenticate users and only allow whitelisted From: email addresses. Step 1: SPF [or similar/better] confirms that the MX is allowed to send email on behalf of the claimed sender address. Discard message if it comes from a bogus MX. Step 2: The MX confirms that the user was authorized to use the claimed sender address. The message would never have been transmitted had the user not authenticated with the trusted MX. Please explain how the trust chain does not verify the sending user. Malware will steal username/password is not a valid answer, as the same can apply equally to crypto keys. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Re: Time to check the rate limits on your mail servers
That, on the other hand, gets you into trouble with rather stupid Spam filters, that only accept mails from a server, if that server is also MX for the senders domain. Yes, this is stupid, but that does not change the fact, that these setups are out there. No, they're not. Large ISPs, starting with AOL and Yahoo, separated their inbound and outbound mail servers years ago. Anyone who still uses mail from MX for filtering doesn't really care if he gets mail or not. Note that this is a different issue from separating your public inbound MX servers from your user-only submit servers. I've done that, too, and haven't had any problems other than educating the occasional too-clever user who thinks my setup instructions must be wrong, substitutes the MX server for the SUBMIT server, and then complains that it doesn't work. Regards, John Levine, [EMAIL PROTECTED], Primary Perpetrator of The Internet for Dummies, Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor I shook hands with Senators Dole and Inouye, said Tom, disarmingly.
Re: Sender authentication zombies (was Re: Time to check the rate limits on your mail servers)
On Sat, 5 Feb 2005, J.D. Falk wrote: DK or IIM makes it clear who is administering the server and this authentication permits reputation assessment. Add an account identifier, and the problem is nailed. Ah, so you're saying that only the reputation of individual e-mail addresses is worth paying attention to? How do you expect that to scale to billions of messages per day? Isn't that called S/MIME and PGP? It hasn't scaled yet. I've received two S/MIME messages in my life, and a few more PGP messages. A problem is if the computer has been compromised, its likely the authentication information stored on the computer has also been compromised or will be when the user starts typing any missing information. Very few consumer-grade computers have advanced security devices installed. As I keep saying, a secure computer rarely DDOSes, spams or sends viruses. And when they do, its much easier to whack the owner. So how do we keep computers secure and fix the insecure ones?
Re: Sender authentication zombies (was Re: Time to check the rate limits on your mail servers)
On Sat, 2005-02-05 at 19:10, J.D. Falk wrote: On 02/05/05, Douglas Otis [EMAIL PROTECTED] wrote: DK or IIM makes it clear who is administering the server and this authentication permits reputation assessment. Add an account identifier, and the problem is nailed. Ah, so you're saying that only the reputation of individual e-mail addresses is worth paying attention to? How do you expect that to scale to billions of messages per day? Without authenticating an identity, it must not be used in a reputation assessment. Currently this is commonly done by using the remote IP address authenticated through the action of transport. In the name space there are two options, the HELO and a validated signature. DK and IIM are attempting to allow the signature solution to scale. -Doug
Re: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005, Edward B. Dreger wrote: JJ auth is sufficient to make email traceable to your own customers. End users also would appreciate the ability to _know_ a message is not forged. The only way to be sure is via cryptographic signature. Barring that level of immediate traceability, SPF provides a very useful data point to that end (as its *only* purpose is curbing forgery). -- -- Todd Vierling [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005, Michael Loftis wrote: --On Thursday, February 03, 2005 11:42 + [EMAIL PROTECTED] wrote: Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Because there are *NO* packages available that offer limiting. Free or commercial. I disagree. On a per IP basis, sendmail now offers ClientRate, number of connections allowed within a 60 second sliding window from a given IP and ClientConn, number of active connections allowed from an IP at any time Used in conjunction with Jochen Bern's bm patch available from http://www.informatik.uni-trier.de/~bern/sendmail/ which limits the number of mail commands given in a single connection, you can rate limit your users fairly well. We have used these limits for ~6 months now and have only had to whitelist 3 sites from the Client limits. You could probably adjust the window size for the ClientRate and then limit the number of smtp commands per connection to achieve like an hourly limit of some sort. sam
RE: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005, Joel Perez wrote: I keep reading these articles and reports about this botnet and that botnet problem and how many user's pc's are infected. The only thing I don't see is a way to remove these bots! http://www.sun.com/software/javadesktopsystem/features.xml http://www.apple.com/macosx/ matto [EMAIL PROTECTED]darwin The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
Re: Time to check the rate limits on your mail servers
On Fri, 2005-02-04 at 09:53 -0500, Todd Vierling wrote: On Thu, 3 Feb 2005, Edward B. Dreger wrote: JJ auth is sufficient to make email traceable to your own customers. End users also would appreciate the ability to _know_ a message is not forged. The only way to be sure is via cryptographic signature. Barring that level of immediate traceability, SPF provides a very useful data point to that end (as its *only* purpose is curbing forgery). Attempting to detect spam trickled through thousands of compromised systems sent through the ISP's mail servers, SPF does nothing, and could actually damage the reputation of those domains that authorize the provider for their mailbox domain using SPF. These records can be read by the spammers and then exploited. Repairing this reputation could be next to impossible. With respect to forgery, authorization is not authentication. There is no consensus which mailbox-domain is checked, SPF (MAILFROM or HELO), Classic (MAILFROM or Other and HELO), or Sender-ID (PRA), so it is uncertain which mailbox-domain may have been checked for authorization, if any. False assurances could be worse than no assurances. White-listing for forwarded accounts or mailing lists to allow an SPF rule-set bypass means there is no certainty a check was ever made. -Doug
Re: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005 11:42:55 +, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top that botnets are now routing their mail traffic through the local ISP's mail servers rather than trying their own port 25 connections. Now? We (and AOL, and some other large networks) have been seeing this thing go on since over a year. Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Doing that - especially now when this article has hit the popular press and there's going to be lots more people doing the same thing - is going to be equivalent of hanging out a block my email sign. One additional thing that I think wasnt mentioned in the article - Make sure your MXs (inbound servers) are separate from your outbound machines, and that the MX servers dont relay email for your dynamic IP netblock. Some other trojans do stuff like getting the ppp domain name / rDNS name of the assigned IP etc and then nslookup -q=mx domain.com, then set itself up so that all its payloads get delivered out of the domain's MX servers -- Suresh Ramasubramanian ([EMAIL PROTECTED])
Re: Time to check the rate limits on your mail servers
Hi! http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top that botnets are now routing their mail traffic through the local ISP's mail servers rather than trying their own port 25 connections. Now? We (and AOL, and some other large networks) have been seeing this thing go on since over a year. Indeed, we also see this a long time now. Most of them specific spamruns towards the bigger players... (AOL alike). Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? One additional thing that I think wasnt mentioned in the article - Make sure your MXs (inbound servers) are separate from your outbound machines, and that the MX servers dont relay email for your dynamic IP netblock. Some other trojans do stuff like getting the ppp domain name / rDNS name of the assigned IP etc and then nslookup -q=mx domain.com, then set itself up so that all its payloads get delivered out of the domain's MX servers So the next article would say 'lets now all seperate MX and SMTP servers' still a LOT of large players combining those two. Giving troyans doing the above scenario a open door. Bye, Raymond.
Re: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005, Suresh Ramasubramanian wrote: On Thu, 3 Feb 2005 11:42:55 +, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top that botnets are now routing their mail traffic through the local ISP's mail servers rather than trying their own port 25 connections. Now? We (and AOL, and some other large networks) have been seeing this thing go on since over a year. Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Doing that - especially now when this article has hit the popular press and there's going to be lots more people doing the same thing - is going to be equivalent of hanging out a block my email sign. I just implemented a patch to tcpserver which allows me to limit the number of simultaneous SMTP connections from any one IP, but have not yet looked into daily/hourly limits. I know Comcast has started limiting residential customers to 50-100 emails per day, and that customers with legitimate reasons for using more than that are starting to complain. One additional thing that I think wasnt mentioned in the article - Make sure your MXs (inbound servers) are separate from your outbound machines, and that the MX servers dont relay email for your dynamic IP netblock. Some other trojans do stuff like getting the ppp domain name / rDNS name of the assigned IP etc and then nslookup -q=mx domain.com, then set itself up so that all its payloads get delivered out of the domain's MX servers Easier said than done, especially if you're a small ISP that's been doing POP before SMTP and changing this requires that every customer's settings be changed. Is there any info on how this zombie is spread? ie, email worms, direct port attacks, etc. If the former, there's hope of nipping it in the bud with anti-virus filtering. James Smallacombe PlantageNet, Inc. CEO and Janitor [EMAIL PROTECTED] http://3.am =
Re: Time to check the rate limits on your mail servers
On Thu, Feb 03, 2005 at 11:42:55AM +, [EMAIL PROTECTED] wrote: CNET reports http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top that botnets are now routing their mail traffic through the local ISP's mail servers rather than trying their own port 25 connections. There is one mistatement in this article, though: the author says: This means the junk mail appears to come from the ISP [...] If it's coming from their servers (or their network), it IS coming from the ISP, and they bear full responsibility for making it stop. ---Rsk
Re: Time to check the rate limits on your mail servers
On Feb 3, 2005, at 9:30 AM, [EMAIL PROTECTED] wrote: One additional thing that I think wasnt mentioned in the article - Make sure your MXs (inbound servers) are separate from your outbound machines, and that the MX servers dont relay email for your dynamic IP netblock. Some other trojans do stuff like getting the ppp domain name / rDNS name of the assigned IP etc and then nslookup -q=mx domain.com, then set itself up so that all its payloads get delivered out of the domain's MX servers Easier said than done, especially if you're a small ISP that's been doing POP before SMTP and changing this requires that every customer's settings be changed. IMHO, if you are a small ISP and limit the # of e-mails per user per day, even to something like 1K, you probably don't have to separate the MX SMTP servers. But that's me, others might still think you were being irresponsible. Is there any info on how this zombie is spread? ie, email worms, direct port attacks, etc. If the former, there's hope of nipping it in the bud with anti-virus filtering. All of the above. -- TTFN, patrick
Re: Time to check the rate limits on your mail servers
Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Doing that - especially now when this article has hit the popular press and there's going to be lots more people doing the same thing - is going to be equivalent of hanging out a block my email sign. I don't understand your comment. This is an arms race. The spammers and botnet builders are attempting to make their bots use the exact same email transmission channels as your customers' email clients. They are getting better at doing this as time goes on. I think we are at the point where the technical expertise of the botnet builders is greater than the technical expertise of most people working in email operations. We cannot win this battle by continuing to attempt to trump their technical abilities. However, if we shift the battleground to a location where network operators have the upper hand, we can do better. And that's why I suggest that people should start looking at email volume controls. The vast majority of individual users only send a small number of emails over a given time period whether you measure that time period in minutes, hours or days. SPAM is a form of DDoS against the Internet's email architecture. Rate limiting has proven to be an effective way of mitigating DDoS because it strikes at the very core of the DoS methodology. Why not deploy this strategy against email? Please note that I am not suggesting that this is a way to solve the SPAM problem. First of all, I do not agree that there is a SPAM problem. The fundamental problem is that the Internet email architecture is flawed. SPAM is merely a symptom of those flaws. If we fix the architecture, then nobody will care about SPAM. As you can see, two separate problems are becoming intertwingled here. In the past we had viruses, DDoS, botnets, SPAM, phishing. But now, all of these things are merging and evolving together. And secondly, I'm only pointing out that there are reasons for people to start thinking about rate limiting email on their networks. I'm suggesting that people should be asking questions. I don't think it is wise to run out and slap rate limits on mail infrastructure without thinking through the implications. --Michael Dillon
Re: Time to check the rate limits on your mail servers
[EMAIL PROTECTED] wrote: CNET reports http://news.com.com/Zombie+trick+expected+to+send+spam+sky-high/2100-7349_3-5560664.html?tag=cd.top that botnets are now routing their mail traffic through the local ISP's mail servers rather than trying their own port 25 connections. Both on ASRG and here on NANOG, many of us said many times, and most of the times people called me crazy; 1. Block port 25 for dynamic ranges - that will kill the current strain of worms. 2. It won't solve spam, and neither will SPF or anything else of the sort, as when you have 100K zombies, you don't need to act a server, you can use the real credentials for the user, and even if limited to a 1000 messages, that times 100K drones is... The issue is numbers, and how to reduce them, not stop the tide. Currently there is a discussion of this on Spam-Research [1], quite interesting. Gadi. 1 - Spam-Research archives: https://linuxbox.org/cgi-bin/mailman/listinfo/spam
Re: Time to check the rate limits on your mail servers
[EMAIL PROTECTED] wrote: On Thu, 3 Feb 2005, Suresh Ramasubramanian wrote: snip Easier said than done, especially if you're a small ISP that's been doing POP before SMTP and changing this requires that every customer's settings be changed. drac http://mail.cc.umanitoba.ca/drac/ supports seperate pop/smtp servers. Which is not neccessarily what is being recommended by having seperate in-mx-smtp and out-smtp. Is there any info on how this zombie is spread? ie, email worms, direct port attacks, etc. If the former, there's hope of nipping it in the bud with anti-virus filtering. James Smallacombe PlantageNet, Inc. CEO and Janitor [EMAIL PROTECTED] http://3.am =
Re: Time to check the rate limits on your mail servers
Hi! Now, once 100K zombies can send *only* 1000 spam messages a day instead of 10K or even 500K, it makes a difference, but it is no solution. I am happy to see people are starting to move this way, and I personally believe that although this is happening (just go and hear what Carl from AOL says on Spam-R that they have been seeing since 2003), this is all a POC. We have not yet begun seeing the action. This is no POC, we have seen this happen many many times. Perhaps some drone networks are a little 'behind' but in general, they are perfectly able to do this. Even with some static lists for some large ISPs mailservers they can perfectly initiate it large scale. And yes, it does limit, but with the number of bots we see controlled on the few botnets we monitored the impact will still be hudge. Should I once again be stoned, or will others see it my way now that the tide is starting to turn? Its not turning, its happening. Bye, Raymond.
Re: Time to check the rate limits on your mail servers
If a pro cannot clean it out safely, then i cannot imagine our typical homeuser would be able to... and with some luck he installs a firewall and antivirus next time, after reinstalling his system for the 4th or 5th time. You may want to check out some AT (Anti-Trojan) software such as The Cleaner and BOclean. Gadi.
Re: Time to check the rate limits on your mail servers
Hi! If a pro cannot clean it out safely, then i cannot imagine our typical homeuser would be able to... and with some luck he installs a firewall and antivirus next time, after reinstalling his system for the 4th or 5th time. You may want to check out some AT (Anti-Trojan) software such as The Cleaner and BOclean. You will never be sure you have picked up all, only the known ones. For a compromised system, unless running tripwire or something, reinstall! Its a nice start, but it also tell people i am safe, and they dont know for sure. Seeing our abuse department getting tickets over and over about the same customers its a fact that they just simple are not able to clean it out easilly. Then its better to instert foot (CD) and start all over. Bye, Raymond
Re: Time to check the rate limits on your mail servers
You will never be sure you have picked up all, only the known ones. For a compromised system, unless running tripwire or something, reinstall! You can never be sure, that's why it's a backdoor/Trojan horse. Its a nice start, but it also tell people i am safe, and they dont know Yes, it is. AV products have not taken Trojan horses seriously for years, and called them garbage samples. Now they start to change that due to almost any sample out there being also a Trojan horse, but not drastically enough for sure. Seeing our abuse department getting tickets over and over about the same customers its a fact that they just simple are not able to clean it out easilly. Then its better to instert foot (CD) and start all over. Then using AT programs is a good start. A clean slate is always better, but your grandma won't agree. Gadi.
Re: Time to check the rate limits on your mail servers
Now, once 100K zombies can send *only* 1000 spam messages a day instead of 10K or even 500K, it makes a difference, but it is no solution. I'd like to see rate limits set much lower than that. Perhaps one message per day to begin with. After the message is sent, send the customer a reminder about the limit and tell them how to get to a web page to increase the limit. The web page would only accept an incremental increase. For instance, if your limit is one, you can bump it up to five per day and that is all. Then, if you exceed the new limit, you once again have the opportunity to bump it up by five more. Most people won't need more than 10 or 15 per day limits. People who need more can call their customer representative and order the volume mail add-on product. They will have to agree to a contract that allows you, the operator, to completely block their net access without notice if it appears that a bot/virus may have infected their systems. I'm sure if you discuss this kind of stuff with your product development and product marketing people, they will come up with more interesting variations. One message per day is not too low. There are people who never use email. They just browse the web and use IM. Why should you, the operator, allow those customers to inject huge numbers of email systems into the Internet as botnet drones? 1000 a day is way too high, IMHO. --Michael Dillon
Re: Time to check the rate limits on your mail servers
This is no POC, we have seen this happen many many times. Perhaps some Wrong, and I will tell you why in a second. drone networks are a little 'behind' but in general, they are perfectly able to do this. Even with some static lists for some large ISPs mailservers they can perfectly initiate it large scale. And yes, it does limit, but with the number of bots we see controlled on the few botnets we monitored the impact will still be hudge. You have been seeing them try it, yes. But why should they use it when they can send 10,000,000,000 spam messages out with no trouble? The answer is because they will soon have to. As much as some are capable of it, most are not yet there. They will be soon. This is the first evolutionary step I can see that we pushed the spammers into doing, according to our wishes. It may be a bigger attack on your servers, but it's nothing in comparison to spam messages out there where every available host sends the spam out. Why SPF won't work? Why it is all useless (SPF, etc.) is because there are 100K and more drone armies out there, but don't kid yourselves - you ain't seen nothing yet. Should I once again be stoned, or will others see it my way now that the tide is starting to turn? Its not turning, its happening. You will know when it's happening. That will be when every spammer will be at the corner and will have to move to this way of working. Just because you see a POC and some people are either more adavanced or bored to do it, and spam is a massive thing so you feel it, doesn't mean it's a trend. Gadi.
Re: Time to check the rate limits on your mail servers
- Original Message - From: Gadi Evron [EMAIL PROTECTED] Allow me to elaborate; and forget about this article, why limited ourselves? Once big ISP's started blocking port 25/outbound for dynamic ranges, and it finally begun hitting the news, we once again caused the spammers to under-go evolution. In this particular case, they figured they'd have to find better ways to send spam out, because eventually, they will be out of working toys. Hello I am a bit concerned that blocking any port at all preventing abuse of the affected service will make the abusers go through other services instead. Port 139/445 is already blocked by several isps due to excessive abuse or I believe they call it 'a security measurement'. Even port 23 has been blocked (inbound and outbound) by atleast 1 large isp I am aware of. When that mssql worm was lurking around isps were also blocking that port. I hope I'm not the only one seeing a pattern here. Really, blocking ports makes no sense to me in the long run. You are destroying the service, and even if you block all ports there are several ways to spam anyway. You would probably reply now saying that yeah but you get rid of 99% of the spammers that way. That is only partly true. As time goes on all spammers will adopt to your isps new security policy and if you still don't see the pattern I am talking about now there is nothing more I can say. I don't have the solution to all of this, but I sure know how to see what is not the solution. Teach people how to write Hello world better perhaps. Joergen Hovland Joergen Hovland ENK
Re: Time to check the rate limits on your mail servers
I know that I'm in the middle of trying to figure this out with the mail server software that is used where I work but if limits are going to be put into place per email box of say 1,000 messages per day and a total daily sending limit of say 200 megabytes, I feel there also needs to be methods in place for the end-user (customer) to be able to view where they stand in relationship to their quota. Yes this becomes more of something for the help desk side of a provider but as operations, I have to support the help desk in being able to give the user information when they call about the limits David - Original Message - From: Gadi Evron [EMAIL PROTECTED] To: Raymond Dijkxhoorn [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; nanog@merit.edu Sent: Thursday, February 03, 2005 10:14 AM Subject: Re: Time to check the rate limits on your mail servers Did you actially read the article? This was about drones sending out via its ISP mailserver. Blocking outbound 25 doesnt help a bit here. In general sure, good ide, and also start using submission for example. But in this contect its silly. No, it is relevant or I wouldn't have mentioned it. Allow me to elaborate; and forget about this article, why limited ourselves? Once big ISP's started blocking port 25/outbound for dynamic ranges, and it finally begun hitting the news, we once again caused the spammers to under-go evolution. In this particular case, they figured they'd have to find better ways to send spam out, because eventually, they will be out of working toys. Using the user's own mail server, whether by.. erm.. just utilizing it if that is possible, sniffing the SMTP credentials or stealing them from a file/registry, maybe even using Outlook to send is all that's about to happen. heck, I don't see how SMTP auth would help, either. They have local access to the machine. Now, once 100K zombies can send *only* 1000 spam messages a day instead of 10K or even 500K, it makes a difference, but it is no solution. I am happy to see people are starting to move this way, and I personally believe that although this is happening (just go and hear what Carl from AOL says on Spam-R that they have been seeing since 2003), this is all a POC. We have not yet begun seeing the action. Should I once again be stoned, or will others see it my way now that the tide is starting to turn? Gadi.
Re: Time to check the rate limits on your mail servers
Hello I am a bit concerned that blocking any port at all preventing abuse of the affected service will make the abusers go through other services instead. Port 139/445 is already blocked by several isps due to excessive abuse or I believe they call it 'a security measurement'. Even port 23 has been blocked (inbound and outbound) by atleast 1 large isp I am aware of. When that mssql worm was lurking around isps were also blocking that port. I hope I'm not the only one seeing a pattern here. Really, blocking ports makes no sense to me in the long run. You are destroying the service, and even if you block all ports there are several ways to spam anyway. You would probably reply now saying that yeah but you get rid of 99% of the spammers that way. That is only partly true. As time goes on all spammers will adopt to your isps new security policy and if you still don't see the pattern I am talking about now there is nothing more I can say. I don't have the solution to all of this, but I sure know how to see what is not the solution. Teach people how to write Hello world better perhaps. I quite agree, blocking ports is not the best answer, as it is a self-inflicted-DDoS. Still, please tell me, how is not blocking un-used or un-necessary ports a bad thing? It is a defensive measure much like you'd add barricades before an attack. The Internet is a war zone, but I don't have to tell the NANOG community that. Thing is, blocking port 25 won't cause spam to stop, there are no FUSSP solution. Yet, we all recognize that SMTP is far from perfect. And indeed, as others here are more qualified than me, by far, to tell you, most development in anti-spam technology only helped short-term, and caused the bad guys to evolve. Well, why is blocking port 25 different? See for yourself. They now evolved, and are using user-credentials and ISP-servers. This evolution means that their capabilities are severely decreased, at least potentially. This is the best next thing after dark Irish stout and ketchup. It means ISP's will have to re-think their strategies, just like AOL did. It also means it's once small step to victory for us. We are a long way from it, and please - not everybody blocks port 25 so current-day worms are more than efficient still. It is nice to see fore thinking and long-term planning with the bad guys, where all we can do is disagree. Gadi.
Re: Time to check the rate limits on your mail servers
On Thu, 03 Feb 2005 16:07:10 +0100, Raymond Dijkxhoorn said: The only thing I don't see is a way to remove these bots! Not everyone knows how to even look at their machines for signs of these bots. Heck, I know most of my guys here don't even know how these bots work. For a compromised system, insert CD, reinstall! BZZT! But thank you for playing. Don't *RE*-install. If you got whacked by a bot on Monday, and re-install Sunday's configuration of software on Tuesday, all that means is that Wednesday you'll get re-whacked. Lather, rinse, repeat. Install *SOMETHING ELSE*. Something less vulnerable to all this manure. (I'll mention the *other* alternative, replacing/upgrading the user, mostly for completeness and so we can all have a good chuckle) pgpLVSBP3QLeD.pgp Description: PGP signature
Re: Time to check the rate limits on your mail servers
On Thu, 03 Feb 2005 17:54:28 +0200, Gadi Evron [EMAIL PROTECTED] wrote: Still, please tell me, how is not blocking un-used or un-necessary ports a bad thing? It is a defensive measure much like you'd add barricades before an attack. Agreed. And depending on your service, there are different ports worth blocking. For residential users, I can't see a reason to not block something like Netbios. And blocking port 25 effectively prevents zombies from spamming. Unfortunately, it also blocks legitimate users from being able to use SMTP AUTH on a remote server.. They now evolved, and are using user-credentials and ISP-servers. This evolution means that their capabilities are severely decreased, at least potentially. Has this been confirmed? Does this new worm, in fact, use SMTP AUTH where necessary? Will it also check the port that the user's computer is set to send mail on? So, for instance, if SMTP AUTH is required, and the mail submission port is being used rather than standard port 25, will the worm detect all this? The nice part about SMTP AUTH, though, is that there is at least a direct link to the user sending the spam. This means, of course, that ISP's will need to police their users a little better.. :) It means ISP's will have to re-think their strategies, just like AOL did. It also means it's once small step to victory for us. We are a long way from it, and please - not everybody blocks port 25 so current-day worms are more than efficient still. So I guess users will have to stop clicking that Save Password button... That is, until the worm records the keystrokes when the password is entered... *sigh* Gadi. -- Jason 'XenoPhage' Frisvold [EMAIL PROTECTED]
Re: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005, Raymond Dijkxhoorn wrote: One additional thing that I think wasnt mentioned in the article - Make sure your MXs (inbound servers) are separate from your outbound machines, and that the MX servers dont relay email for your dynamic IP netblock. Some other trojans do stuff like getting the ppp domain name That, on the other hand, gets you into trouble with rather stupid Spam filters, that only accept mails from a server, if that server is also MX for the senders domain. Yes, this is stupid, but that does not change the fact, that these setups are out there. Start using authenticated SMTP for this. Until the next bot implemented co-opts the pop3 client, or simply hacks the password from the pop3 client (how strong is that encryption?). James Smallacombe PlantageNet, Inc. CEO and Janitor [EMAIL PROTECTED] http://3.am =
Re: Time to check the rate limits on your mail servers
On Thu, 03 Feb 2005 12:16:41 EST, Jason Frisvold said: Agreed. And depending on your service, there are different ports worth blocking. For residential users, I can't see a reason to not block something like Netbios. And blocking port 25 effectively prevents zombies from spamming. Unfortunately, it also blocks legitimate users from being able to use SMTP AUTH on a remote server.. There's a *reason* why RFC2476 specifies port 587 pgpGfzZdCd46O.pgp Description: PGP signature
Re: Time to check the rate limits on your mail servers
On Thu, Feb 03, 2005 at 05:29:15PM +0200, Gadi Evron wrote: You will never be sure you have picked up all, only the known ones. For a compromised system, unless running tripwire or something, reinstall! You can never be sure, that's why it's a backdoor/Trojan horse. Its a nice start, but it also tell people i am safe, and they dont know Yes, it is. AV products have not taken Trojan horses seriously for years, and called them garbage samples. Now they start to change that due to almost any sample out there being also a Trojan horse, but not drastically enough for sure. Seeing our abuse department getting tickets over and over about the same customers its a fact that they just simple are not able to clean it out easilly. Then its better to instert foot (CD) and start all over. Then using AT programs is a good start. A clean slate is always better, but your grandma won't agree. Unfortunately, starting over in some operating systems means re-installing EVERYTHING, and since applications tend to get installed over time, the installation media for each and every app may not be available. Backups are not very useful, because just placing the executables and the work product/data files in the right place will not work in some Windows systems if the proper registry entries are not there. Also, if you reinstall in the wrong order you can wind up in DLL hell. Gadi. -- -=[L]=-
Re: Time to check the rate limits on your mail servers
On 02/03/05, [EMAIL PROTECTED] wrote: Is there any info on how this zombie is spread? ie, email worms, direct port attacks, etc. If the former, there's hope of nipping it in the bud with anti-virus filtering. Yeah, that's been working really well for us so far. /sarcasm -- J.D. Falk uncertainty is only a virtue [EMAIL PROTECTED]when you don't know the answer yet
Re: Time to check the rate limits on your mail servers
on Thu, Feb 03, 2005 at 04:07:10PM +0100, Raymond Dijkxhoorn wrote: The only thing I don't see is a way to remove these bots! Not everyone knows how to even look at their machines for signs of these bots. Heck, I know most of my guys here don't even know how these bots work. For a compromised system, insert CD, reinstall! ...which simply reinstalls the old vulnerabilities that made the machine suspectible to compromise in the first place. If you can't patch up from the buggy baseline in time, reinstalling from original media is often the worst thing you can do, if the machine is still connected to the network. And if the machine is NOT connected to the network, it is often not possible to get the security updates downloaded that patch the vulnerabilities. -- hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com join us! http://hesketh.com/about/careers/account_manager.htmljoin us!
Re: Time to check the rate limits on your mail servers
GE Date: Thu, 03 Feb 2005 17:14:40 +0200 GE From: Gadi Evron GE heck, I don't see how SMTP auth would help, either. They have local GE access to the machine. User joe6pack is pumping out 100k messages/day. That can't possibly be valid; let's disable his -- and only his -- SMTP access. He can't spam directly via SMTP/25 connections, so we're good there. User joe6pack's mail volume is two sigma above normal. Good thing our outbound mail spam scanning is much more stringent under these conditions. User joe6pack doesn't know which of 50 machines behind his SOHO's NAT box sent the spam. Luckily, the username helps us/him track down the infected box. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Re: Time to check the rate limits on your mail servers
GE Date: Thu, 03 Feb 2005 17:54:28 +0200 GE From: Gadi Evron GE They now evolved, and are using user-credentials and ISP-servers. This GE evolution means that their capabilities are severely decreased, at least GE potentially. This means that it's 1998 again. Direct-to-MX spam was an evolution when user accounts began getting nuked for spamming. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Re: Time to check the rate limits on your mail servers
: I'd like to see rate limits set much lower than that. Perhaps one : message per day to begin with. After the message is sent, send the : customer a reminder about the limit and tell them how to get to a web : page to increase the limit. The web page would only accept an : incremental increase. For This is a great way to attract and keep customers. I also like the other's suggestions that cause their customers a lot of hassle and pain. As some folks have said before, I encourage my competition to do this. : services instead. Port 139/445 is already blocked by several isps due : to excessive abuse or I believe they call it 'a security measurement'. NetBIOS was never meant to be a WAN protocol. For your customers that need this so they can share files, it's a dangerous thing because folks with that level of expertise wouldn't know how to protect their personal data. There is no need to let NetBIOS ports be open to the world. : atleast 1 large isp I am aware of. When that mssql worm was lurking : around isps were also blocking that port. I hope I'm not the only one You want your MySQL database open to the world??? What's you IP address? Never mind, I could find it anyway... 8-) : seeing a pattern here. Really, blocking ports makes no sense to me in : the long run. Again, some protocols were never meant to be available to the world, so there is a need to block some things. Some should be restricted to the local LAN, some should be restricted to your network, or some part of it, and some open to the world. scott
Re: Time to check the rate limits on your mail servers
On Thu, 03 Feb 2005 12:26:55 -0500, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: On Thu, 03 Feb 2005 12:16:41 EST, Jason Frisvold said: Agreed. And depending on your service, there are different ports worth blocking. For residential users, I can't see a reason to not block something like Netbios. And blocking port 25 effectively prevents zombies from spamming. Unfortunately, it also blocks legitimate users from being able to use SMTP AUTH on a remote server.. There's a *reason* why RFC2476 specifies port 587 I assume you're referring to the ability to block port 25 if 587 is used for submission. This is great in theory, but if this were the case, then the Trojan authors would merely alter their Trojan to use port 587. Unfortunately, I don't think there's an easy answer to the spam problem. Sure, we can educate and block. But at the end of the day, the spammers will just find another way to worm those messages into the network. Some of these guys are making boatloads of money, and I hardly think they're willing to throw in the towel if they hit a bump in the road... On the flipside, those of us working as admins and trying to stop the flow of spam are making next to nothing.. *sigh* -- Jason 'XenoPhage' Frisvold [EMAIL PROTECTED]
Re: Time to check the rate limits on your mail servers
On Thu, Feb 03, 2005 at 12:26:55PM -0500, [EMAIL PROTECTED] wrote: On Thu, 03 Feb 2005 12:16:41 EST, Jason Frisvold said: Agreed. And depending on your service, there are different ports worth blocking. For residential users, I can't see a reason to not block something like Netbios. And blocking port 25 effectively prevents zombies from spamming. Unfortunately, it also blocks legitimate users from being able to use SMTP AUTH on a remote server.. There's a *reason* why RFC2476 specifies port 587 IIRC the starting point of this thread was, that Spammers now learned to use the smarthost of the clients. When they are using that, why is it more difficult for them to send their junk on port 587 instead of port 25? As soon as the spammers on a big scale learn to use the same traffic path the mailclients do, instead looking up MXes themselves, this switching ports and blocking 25 that is proposed, will cause a lot of work without any benefit. Same goes for SPF, BTW. Only thing that puzzles me is, why it took spammers so long to go in this direction. Nils
Re: Time to check the rate limits on your mail servers
--On Thursday, February 03, 2005 11:42 + [EMAIL PROTECTED] wrote: Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Because there are *NO* packages available that offer limiting. Free or commercial.
Re: Time to check the rate limits on your mail servers
Nils Ketelsen wrote: Only thing that puzzles me is, why it took spammers so long to go in this direction. It didn't. It took the media long to notice. Pete
Re: Time to check the rate limits on your mail servers
- Original Message - From: Jason Frisvold [EMAIL PROTECTED] On Thu, 03 Feb 2005 17:54:28 +0200, Gadi Evron [EMAIL PROTECTED] wrote: Still, please tell me, how is not blocking un-used or un-necessary ports a bad thing? It is a defensive measure much like you'd add barricades before an attack. Agreed. And depending on your service, there are different ports worth blocking. For residential users, I can't see a reason to not block something like Netbios. And blocking port 25 effectively prevents zombies from spamming. Unfortunately, it also blocks legitimate users from being able to use SMTP AUTH on a remote server.. I still can't really agree. How do you know a port is un-used or un-necessary? Because IANA has assigned port 25 as SMTP? Because only crackers use netbios outside their lan? You can't really inspect your network for a month to determine what ports are being used legit either since this changes over time and the list of ports would be noisy due to virus' etc. And why should you block that particular port when there are no difference between port numbers technically speaking? The only valid reason would be because the other party is also using that port and blocking that particular port will prevent that particular traffic unless somebody changed the portnumber - which will happen if you start blocking specific ports because it might just annoy certain people too much. This is why all the socket enabled software we develop always use port 80 or 443 to be able to get through firewalls. We simply don't want to spend the extra time helping and telling the customer to enable this and that port on their firewall. So in 20 years when every single program is using the same port because you are blocking all the other ports - how can you tell the difference? Packet inspection! But no not always, not when you are using SSL etc. Oh okay, then lets disable that then since you can't identify those packets and because we don't care about the collateral damage it gives anyway? To a solution I would consider okay: Since port 25 is mostly known as belonging to SMTP I would rather transparently proxy all outbound 25 connections from customers to our outbound SMTP server instead of blocking the port directly. If the proxy was unable to detect that this was a legit SMTP connection, it will redirect to the original target instead. Now, what will happen is that your companies SMTP server will catch every single bot/worm spamming through SMTP. Here is when the rate-limit and outbound spam/virusfilters should kick in. If you were sending more than 10 infected e-mails or you are actually spamming (yourself or not), disable the customers internet connectivity and redirect port 80 requests to an information page telling the customer you are infected, click here to download antivirus etc... and click here when you think you have removed the virus/stopped spamming to regain full connectivity. Virus' could automaticly detect this so you shouldn't make it too easy to regain internet access. This would help your customer finding out if their equipment is infected instead of being unaware of it (since you block port 25 instead). If the customers laptop was infected and he/she frequently moves to other isps (wlan etc) not blocking that port, it could be harder to find out for both parties. They now evolved, and are using user-credentials and ISP-servers. This evolution means that their capabilities are severely decreased, at least potentially. Has this been confirmed? Does this new worm, in fact, use SMTP AUTH where necessary? Will it also check the port that the user's computer is set to send mail on? So, for instance, if SMTP AUTH is required, and the mail submission port is being used rather than standard port 25, will the worm detect all this? The nice part about SMTP AUTH, though, is that there is at least a direct link to the user sending the spam. This means, of course, that ISP's will need to police their users a little better.. :) It means ISP's will have to re-think their strategies, just like AOL did. It also means it's once small step to victory for us. We are a long way from it, and please - not everybody blocks port 25 so current-day worms are more than efficient still. So I guess users will have to stop clicking that Save Password button... That is, until the worm records the keystrokes when the password is entered... *sigh* Gadi. -- Jason 'XenoPhage' Frisvold [EMAIL PROTECTED] Joergen Hovland Joergen Hovland ENK
Re: Time to check the rate limits on your mail servers
Nils Ketelsen wrote: Only thing that puzzles me is, why it took spammers so long to go in this direction. Nils I am still confused why people think this is new behavior. The sky is not falling (regardles of how many stories CNET publishes claiming it is), nor should this really be relevant to how I operate my network. This is purely a systems administration issue to tackle, which I believe is beyond the scope of this list. I do find it amazing that we cannot go more than a month without raising some spam-related thread and beating it to death. Andy
Re: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005, Jason Frisvold wrote: prevents zombies from spamming. Unfortunately, it also blocks legitimate users from being able to use SMTP AUTH on a remote server.. There's a *reason* why RFC2476 specifies port 587 I assume you're referring to the ability to block port 25 if 587 is used for submission. This is great in theory, but if this were the case, then the Trojan authors would merely alter their Trojan to use port 587. If they authenticate. Modulo a stupidity built-in to Sendmail (that Claus Assman ignorantly thinks is a non-issue[*]), port 587 is not supposed to be used for endpoint MTA delivery. It's a mail SUBMISSION port, which is supposed to mean that J. Random Client isn't supposed to use it for delivery purposes. === [*] As of now, Sendmail doesn't require one of SMTP AUTH auth by default on the MSA port; it treats 25 and 587 identically (so that things like IP-based relay auth work without need for SMTP AUTH). I sent a m4-only change to the Sendmail maintainers implementing a way to make 587 allow only relay-authorized clients to send anything at all by default -- whther IP-based relay auth, or SMTP AUTH, or any other method built in to the relay-check code path. It was shot down by Claus because he simply doesn't understand the issue and doesn't think identical 25 and 587 ports is a threat. -- -- Todd Vierling [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: Time to check the rate limits on your mail servers
Michael Loftis wrote: Because there are *NO* packages available that offer limiting. Free or commercial. Strange. Our mail servers have had this ability for over a year. The hard part is getting tens of thousands of legacy ISP customers to switch to SMTP auth without drowning the support center in calls. -- Robert Blayzor, BOFH INOC, LLC rblayzor\@(inoc.net|gmail.com) PGP: http://www.inoc.net/~dev/ Key fingerprint = 1E02 DABE F989 BC03 3DF5 0E93 8D02 9D0B CB1A A7B0 Supercomputer: Turns CPU-bound problem into I/O-bound problem. - Ken Batcher
Re: Time to check the rate limits on your mail servers
Once upon a time, Robert Blayzor [EMAIL PROTECTED] said: Michael Loftis wrote: Because there are *NO* packages available that offer limiting. Free or commercial. Strange. Our mail servers have had this ability for over a year. The hard part is getting tens of thousands of legacy ISP customers to switch to SMTP auth without drowning the support center in calls. What does that have to do with SMTP rate limiting? -- Chris Adams [EMAIL PROTECTED] Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Re: Time to check the rate limits on your mail servers
Chris Adams wrote: What does that have to do with SMTP rate limiting? A lot since the original question was: Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? and an answer was: Because there are *NO* packages available that offer limiting. Free or commercial. So I corrected it, software is available that allows you limit/tarpit SMTP connections as well as limit a number of messages a user can send in a given time period. -- Robert Blayzor, BOFH INOC, LLC rblayzor\@(inoc.net|gmail.com) PGP: http://www.inoc.net/~dev/ Key fingerprint = 1E02 DABE F989 BC03 3DF5 0E93 8D02 9D0B CB1A A7B0 Please excuse me, I have to circuit an AC line through my head to get this database working.
RE: Time to check the rate limits on your mail servers
How come it is always about controlling the symptoms and not the illness? The vast majority of these spam drones are compromised WINDOWS machines. If the operating system and dominant email applications so easily allows the users' machines to be taken over by a third party, then there is something wrong with the operating system and the mail applications. It occurs to me that the solution is not to limit the range of destruction, but to defuse the bomb. Perhaps the focus for a solution should move up the model to layer 7. - Mark -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Thursday, February 03, 2005 8:47 AM To: nanog@merit.edu Subject: Re: Time to check the rate limits on your mail servers Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Doing that - especially now when this article has hit the popular press and there's going to be lots more people doing the same thing - is going to be equivalent of hanging out a block my email sign. I don't understand your comment. This is an arms race. The spammers and botnet builders are attempting to make their bots use the exact same email transmission channels as your customers' email clients. They are getting better at doing this as time goes on. I think we are at the point where the technical expertise of the botnet builders is greater than the technical expertise of most people working in email operations. ...
Re: Time to check the rate limits on your mail servers
Miller, Mark wrote: How come it is always about controlling the symptoms and not the illness? The illness is the user. That is uncontrollable.
Re: Time to check the rate limits on your mail servers
On 02/03/05, Miller, Mark [EMAIL PROTECTED] wrote: How come it is always about controlling the symptoms and not the illness? The vast majority of these spam drones are compromised WINDOWS machines. If the operating system and dominant email applications so easily allows the users' machines to be taken over by a third party, then there is something wrong with the operating system and the mail applications. It occurs to me that the solution is not to limit the range of destruction, but to defuse the bomb. Perhaps the focus for a solution should move up the model to layer 7. Upgrading and/or replacing the OS for every Windows user on the planet is an educational issue. Keeping the network viable while you figure out how to do that is an operational issue. -- J.D. Falk uncertainty is only a virtue [EMAIL PROTECTED]when you don't know the answer yet
Re: Time to check the rate limits on your mail servers
Michael Loftis [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Because there are *NO* packages available that offer limiting. Free or commercial. My exim.conf calls you a liar. -- Madam, there's no such thing as a tough child - if you parboil them first for seven hours, they always come out tender. - W.C. Fields
Re: Time to check the rate limits on your mail servers
On Thu, Feb 03, 2005 at 09:21:19PM +0200, Petri Helenius wrote: Nils Ketelsen wrote: Only thing that puzzles me is, why it took spammers so long to go in this direction. It didn't. It took the media long to notice. Pete's correct. And there's another reason: spammers have long since demonstrated that they will adapt when necessary. Now that some ISPs have FINALLY, more than two years after they were warned that they needed block port 25 inbound/outbound ASAP on as much of their address space as possible in order to put a sock in this, done something...the spammers may have judged that it's become necessary. And please note: this is far, FAR from the last thing that they have in their bag of tricks. ---Rsk
Re: Time to check the rate limits on your mail servers
Creating an invincible mail client, still only addresses the symptom, and not the disease. I would contend that any attempts made to harden a mail client, will, (and have always been..), be countered with a new exploit, a new method of exploiting the system. The only way to really control spam, is to make it unprofitable, both for the hosting providers, and websites that use this as a form of mass marketing. If say, a 'top 100 domains' (or 10,000, if need be..), list of offending websites were assembled, continually updated, and used universally to null route the websites paying for these services, (and in some cases, entire blocks owned by unscrupulous service providers hosting these websites, in the case that are continually proffering these services to offending parties..), it would soon become the case that if you use spam to mass market your product, you risk losing your access to a portion of the internet. Of course, there are many lists of this kind, but what is lacking, is the willingness to launch a coordinated effort, or agreement on a proven and effective criteria for identifying how this could/should be regulated. I have heard the argument that we are not in the business of determining what should be permitted on the internet, and for the most part I would tend to agree, but I view this as a technical and not an ethical issue, and when seen in that context, the solutions seem obvious. Control spam? Attack it at the source, -follow the money- and make those that would profit from the abuse of the system accountable by denying them services. John - Original Message - From: Miller, Mark [EMAIL PROTECTED] To: nanog@merit.edu Sent: Thursday, February 03, 2005 3:37 PM Subject: RE: Time to check the rate limits on your mail servers How come it is always about controlling the symptoms and not the illness? The vast majority of these spam drones are compromised WINDOWS machines. If the operating system and dominant email applications so easily allows the users' machines to be taken over by a third party, then there is something wrong with the operating system and the mail applications. It occurs to me that the solution is not to limit the range of destruction, but to defuse the bomb. Perhaps the focus for a solution should move up the model to layer 7. - Mark -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Thursday, February 03, 2005 8:47 AM To: nanog@merit.edu Subject: Re: Time to check the rate limits on your mail servers Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Doing that - especially now when this article has hit the popular press and there's going to be lots more people doing the same thing - is going to be equivalent of hanging out a block my email sign. I don't understand your comment. This is an arms race. The spammers and botnet builders are attempting to make their bots use the exact same email transmission channels as your customers' email clients. They are getting better at doing this as time goes on. I think we are at the point where the technical expertise of the botnet builders is greater than the technical expertise of most people working in email operations. ...
Re: Time to check the rate limits on your mail servers
We've been doing this on postfix for some time now. Michael Loftis wrote: --On Thursday, February 03, 2005 11:42 + [EMAIL PROTECTED] wrote: Do you let your customers send an unlimited number of emails per day? Per hour? Per minute? If so, then why? Because there are *NO* packages available that offer limiting. Free or commercial.
RE: Time to check the rate limits on your mail servers
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of J.D. Falk Sent: Thursday, February 03, 2005 4:35 PM To: nanog@merit.edu Subject: Re: Time to check the rate limits on your mail servers On 02/03/05, Miller, Mark [EMAIL PROTECTED] wrote: How come it is always about controlling the symptoms and not the illness? The vast majority of these spam drones are compromised WINDOWS machines. If the operating system and dominant email applications so easily allows the users' machines to be taken over by a third party, then there is something wrong with the operating system and the mail applications. It occurs to me that the solution is not to limit the range of destruction, but to defuse the bomb. Perhaps the focus for a solution should move up the model to layer 7. Upgrading and/or replacing the OS for every Windows user on the planet is an educational issue. Keeping the network viable while you figure out how to do that is an operational issue. ..or a cost issue. Most of these users are people who have decided not to spend the $40 to defend their machine at home. -M
Re: Time to check the rate limits on your mail servers
On 02/03/05, Hannigan, Martin [EMAIL PROTECTED] wrote: Upgrading and/or replacing the OS for every Windows user on the planet is an educational issue. Keeping the network viable while you figure out how to do that is an operational issue. ..or a cost issue. Most of these users are people who have decided not to spend the $40 to defend their machine at home. So you educate them as to why it would be a good idea to keep their computer secure. But in the meantime, their machine is spewing garbage -- which, as many have said, is the operational issue at hand. -- J.D. Falk uncertainty is only a virtue [EMAIL PROTECTED]when you don't know the answer yet
Re: Time to check the rate limits on your mail servers
Peter Corlett [EMAIL PROTECTED] wrote: [...] My exim.conf calls you a liar. Since I've had a few private emails about my rude and abrupt comment (although not complaining about it, which is encouraging :), I'd better explain further, just in case there were people who are curious but not curious enough to email me. Exim4 contains support for executing SQL statements in, for example, PostgreSQL. The original intent was probably so that you can do a SELECT on a PostgreSQL database for performing expansions instead of the more traditional flat files and DBMs/CDBs. However, you can also do an INSERT or UPDATE, which now allows you to maintain state between SMTP transactions. So, to perform rate-limiting, you would create a couple of ACLs: a) A deny ACL that blocks/defers mail submission if a SELECT indicates that the user has exceeded their quota. b) A warn ACL (effectively a no-op as far as access control is concerned) that does an INSERT or UPDATE to increment the user's counter. To identify a user in exim.conf, you can use, for example, their IP address, authenticated username, or some other information available from the SMTP transaction. You can either have a cron job reset the usage counters, or craft your SQL statements so that old counters are ignored. If done right, you would even get counts of daily mail volume for each individual customer in a handy SQL-queriable database for free. -- The only source of knowledge is experience. - Albert Einstein
Re: Time to check the rate limits on your mail servers
How come it is always about controlling the symptoms and not the illness? The illness is the user. That is uncontrollable. A product that doesn't work as advertised has much to do with it as well. Adi
Re: Time to check the rate limits on your mail servers
How about using SMTP AUTH and verifying the envelope MAIL FROM to match the actual user authenticating? This will make SPAM traceable and hopefully ultimately users aware that their PC is sending junk. Adi
Re: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005, Adi Linden wrote: How about using SMTP AUTH and verifying the envelope MAIL FROM to match the actual user authenticating? that doesn't work if you have more than one email address. This will make SPAM traceable and hopefully ultimately users aware that their PC is sending junk. auth is sufficient to make email traceable to your own customers. Adi -- -- Joel Jaeggli Unix Consulting [EMAIL PROTECTED] GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2
Re: Time to check the rate limits on your mail servers
How about using SMTP AUTH and verifying the envelope MAIL FROM to match the actual user authenticating? that doesn't work if you have more than one email address. Wouldn't address resolution take care of that if properly configured? Some implementations allow you to specify what email addresses the user is allowed to send from, that's something that needs to be managed carefully. -GSH
Re: Time to check the rate limits on your mail servers
JJ Date: Thu, 3 Feb 2005 15:41:34 -0800 (PST) JJ From: Joel Jaeggli JJ How about using SMTP AUTH and verifying the envelope MAIL FROM to match JJ the actual user authenticating? JJ JJ that doesn't work if you have more than one email address. The words overreaching and fallacious come to mind. JJ auth is sufficient to make email traceable to your own customers. End users also would appreciate the ability to _know_ a message is not forged. Alas, I doubt much has changed since last October's BCP38 discussions, so perhaps I should not hold my breath. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Re: Time to check the rate limits on your mail servers
On Thu, 2005-02-03 at 14:55 -0800, J.D. Falk wrote: On 02/03/05, Hannigan, Martin [EMAIL PROTECTED] wrote: ..or a cost issue. Most of these users are people who have decided not to spend the $40 to defend their machine at home. So you educate them as to why it would be a good idea to keep their computer secure. But in the meantime, their machine is spewing garbage -- which, as many have said, is the operational issue at hand. Solutions through diligent use of add-on products is not 100%. Many users spend $40 and diligently apply prophylactics, but still are compromised. Reinstalling over an existing installation does not ensure removal. Either way, this returns the OS to a vulnerable state, while costing several frustrating hours. Using a CD-ROM OS/App suite, such as Knoppix, sounds promising for this headache. It should be difficult to corrupt an OS or application when on Read-Only media. :) The number of zombies ensures rate limiting will not be effective either. Providers keeping their house in order in the face of this new strategy may be assisted by domain signed mail. This could serve to block compromised accounts with help from the provider themselves. Rejections from a third party will tell their clients they need a disinfectant. http://mipassoc.org/mass/ The wack-a-mole game needs a more agile mallet. -Doug
Re: Time to check the rate limits on your mail servers
On Thu, 3 Feb 2005 09:30:58 -0500 (EST), [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: I just implemented a patch to tcpserver which allows me to limit the number of simultaneous SMTP connections from any one IP, but have not yet looked into daily/hourly limits. I know Comcast has started limiting residential customers to 50-100 emails per day, and that customers with legitimate reasons for using more than that are starting to complain. See http://spamthrottle.qmail.ca/ for a qmail rate-limiting solution. Setting a limit on the maximum number of messages/minute that will be accepted (and enforcing the limit by tarpitting, by slowing down the server response) seems to be less likely to annoy customers than setting a hard daily quota. One additional thing that I think wasnt mentioned in the article - Make sure your MXs (inbound servers) are separate from your outbound machines, and that the MX servers dont relay email for your dynamic IP netblock. Some other trojans do stuff like getting the ppp domain name / rDNS name of the assigned IP etc and then nslookup -q=mx domain.com, then set itself up so that all its payloads get delivered out of the domain's MX servers. This is a very good suggestion. I also ran into a trojan which would take the target domain name and try to guess mail servers willing to accept mail for the domain by prepending names like mx and smtp and mail1. I ended up renaming mail1 to a more obscure name after noticing that 80% of the blocked worm traffic for a given week was coming in via that one path. At Date: Thu, 3 Feb 2005 09:54:00 -0500 Nils Ketelsen writes: That, on the other hand, gets you into trouble with rather stupid Spam filters, that only accept mails from a server, if that server is also MX for the senders domain. Yes, this is stupid, but that does not change the fact, that these setups are out there. I've set up the outbound and inbound mail servers for many sites, including a Fortune 500 enterprise sending many thousands of messages each day, and have never run into a problem with outbound mail being refused because the outbound mail servers are not listed as an MX for the sender's domain. Not only are the outbound servers not listed in the MX record for the sending domain, but much of the outbound email shows a 'from' address which is a completely different domain than the domain of the server's DNS entry. I don't doubt that there might be sites blocking email based on this criteria, but such a policy is not only shortsighted, but also exceedingly rare. Kevin
Re: Time to check the rate limits on your mail servers
How about using SMTP AUTH and verifying the envelope MAIL FROM to match the actual user authenticating? This will make SPAM traceable and hopefully ultimately users aware that their PC is sending junk. Ouch .. Then spammers may start using a From: matching the SMTP auth user, and effectively joe-jobbing the user.. Ick.. And that would be marvelous! At the very least it would give the user an incentive to clean up his PC. Alternately the email account could be revoked. Adi
Re: Time to check the rate limits on your mail servers
How about using SMTP AUTH and verifying the envelope MAIL FROM to match the actual user authenticating? that doesn't work if you have more than one email address. You should know all your users email addresses. It shouldn't be too difficult to match the 'mail from' address with the user account. The only caveat would be that [EMAIL PROTECTED] would actually have to use the hotmail smtp server to send mail. This will make SPAM traceable and hopefully ultimately users aware that their PC is sending junk. auth is sufficient to make email traceable to your own customers. And how is that? There isn't necessarily anything in an email indicating that it originated from an SMTP AUTH authenticated user. While a header could be added, it isn't a mandatory thing. Adi
Re: Time to check the rate limits on your mail servers
* [EMAIL PROTECTED] (Adi Linden) [Fri 04 Feb 2005, 03:17 CET]: You should know all your users email addresses. You have got to be kidding. -- Niels. -- The idle mind is the devil's playground
Re: Time to check the rate limits on your mail servers
JF Date: Thu, 3 Feb 2005 20:37:29 -0500 JF From: Jason Frisvold JF Ouch .. Then spammers may start using a From: matching the SMTP auth JF user, and effectively joe-jobbing the user.. Ick.. Exactly. The user then loses mail sending ability, but other services remain functional. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Okay i Jeff Bond, I confess. I stole your mail, stalking you day and night
Okay i Jeff Bond, I confess. I stole your mail, stalking you day and night, stole a car and left it in your driveway, making harassing phone calls to you (while my wife screamed at me to stop),and yes, Im cyberstalked you. Oh, and that old grey and red Honda? Yeah, that's mine too (in addition to the White Bronco like OJ's). TAHOEZBOXMAN PLES. EMAIL ME FOR VIDEO W/SOUND AND PICS.
Re: your mail
This is a topic I get very soap-boxish about. I have too many problems with providers who don't understand the college student market. I can think of one university who requires students to login through a web portal before giving them a routable address. This is such a waste of time for both parties. Sure it makes tracking down the abusers much easier, but is it worth the time and effort to manage? This is a very legitimate idea for public portals in common areas, but not in dorm rooms. I've been offline for a few days and I'm catching up, so I might be taking this one out of context. If so, I'm sure I'll be flamed appropriately. The University that I work for has one of these go to a web page and authenticate to get a valid IP though, admittedly, we only make them authenticate once. What does it take to manage? Just the up front work to put the system in place (which wasn't much). For the small inconvenience of logging in once and the extremely small overhead in maintaining the system, we've found a log of uses. Two examples come to mind. We have the ability to automate the forwarding of DMCA violation notices because we know what human was responsible for the offense that occured a few weeks/months back. We also have the ability to contact a human when their system is infected instead of merely shutting their port, waiting for them to call, and hoping that our help desk correlates the my computer isn't working with the this port is shut for a security incident. We might know what dorm room the computer is in, but our rooms sometimes have four people with four to six computers and almost none of our students use their land-line, opting for a cell phone that's not listed in the campus directory... Anyway, knowing what room the computer is in really doesn't provide us much help unless we want someone to walk over there. With a username, we can at least send them an email or put them on a watch list for when they call Eric :)
Re: your mail
On Wed, 5 Feb 2003, Lynn Bashaw wrote: Does anyone on the list know of any ISPs that bill based on average utilization, rather than some variation of 95th percentile? Average is just a function of total and time, and time progresses linearly with time, so average x some $ figure is just the same as saying total x some other $ figure. So most people would just look at that as being billing based upon total traffic volume, which yes, there are folks who do. I don't know of any in the U.S., but it's very common overseas, particularly places which have satellite links in their upstream path. -Bill
RE: your mail
On Wed, 5 Feb 2003, Koepp, Karsten wrote: Volume usually totals in+out, whereas average does max(in,out) divided by time intervals. Well, not to be nit-picky, but that wouldn't strictly be averaging, then. To get back to the question at hand, another scheme that I'm seeing more prevalent lately is dual-bin, where local traffic is flat-rate, and international traffic is measured-rate (i.e. billed per-bit). That's relatively easy to do in a simple satcom-up sort of network, since you just bill by netflow accounting on the satcom interface. There's been a really good discussion of these sorts of billing methods going on on the NordNOG list over the past two weeks, and we'll be having a panel discussion on billing methods at the NordNOG meeting next week. -Bill
Re: your mail
Does anyone on the list know of any ISPs that bill based on average utilization, rather than some variation of 95th percentile? Sure. As long as your math is correct it does not matter how do you calculate your bill. Alex
Re: your mail
My thoughts are Cogents primary customers are sites that are looking for very cheap bandwidth, which most likely is adult content. Therefore they would look more like a content provider than a transit provider. Cogent is making in roads at a lot of Universities who want, as we all know, large amounts bandwidth but don't want to pay for it :) Whether the University is a sink or source of traffic typically comes down to whether or not they filter/rate-limit their peer-to-peer traffic. If they do, then the University will look like any traditional end-user ISP. If they don't, then the University will like like a hosting provider with lots of content... Eric :)
Re: your mail
At 12:32 AM 8/21/2002 -0400, David Lesher wrote: Unnamed Administration sources reported that N. Richard Solis said: If you haven't worked in an environment where you had to turn in your cellphone and pager at the front desk, show a badge to a camera around every corner, and get your office keys from a vending machine you dont know what real security looks like. You missed the places w/ real security. That's where the very polite Marine Security Guard with the 870 shotgun asks to see your badge again... Can we all stop talking shit for a moment? Real security is when nobody can talk about it. Regards, -- Martin Hannigan[EMAIL PROTECTED]
Re[2]: your mail
On Wed, 21 Aug 2002 00:32:24 -0400 (EDT) David Lesher [EMAIL PROTECTED] wrote: Unnamed Administration sources reported that N. Richard Solis said: If you haven't worked in an environment where you had to turn in your cellphone and pager at the front desk, show a badge to a camera around every corner, and get your office keys from a vending machine you dont know what real security looks like. You missed the places w/ real security. That's where the very polite Marine Security Guard with the 870 shotgun asks to see your badge again... or you're standing in the parking lot, and suddenly find yourself surrounded by men in suits carrying mac-10s. richard -- Richard Welty [EMAIL PROTECTED] Averill Park Networking 518-573-7592 Unix, Linux, IP Network Engineering, Security
RE: your mail
Who did you think held the cellphone and the pager? :-) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of David Lesher Sent: Wednesday, August 21, 2002 12:32 AM To: nanog list Subject: Re: your mail Unnamed Administration sources reported that N. Richard Solis said: If you haven't worked in an environment where you had to turn in your cellphone and pager at the front desk, show a badge to a camera around every corner, and get your office keys from a vending machine you dont know what real security looks like. You missed the places w/ real security. That's where the very polite Marine Security Guard with the 870 shotgun asks to see your badge again... -- A host is a host from coast to [EMAIL PROTECTED] no one will talk to a host that's close[v].(301) 56-LINUX Unless the host (that isn't close).pob 1433 is busy, hung or dead20915-1433
RE: Shared facilities (was Re: your mail)
Sean, For a lot of people, these locations are a place to store an entire web presence. That might include order information or private email or credit card records for an entire day's transactions. My feeling is that the general purpose of security at these locations is to make sure that no one is tampering with any equipment in any way, to include unauthorized removal. That was the point of my previous email. The connections to those machines and the data stored on them is what is of value in those locations, not the physical security of the people. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Sean Donelan Sent: Wednesday, August 21, 2002 2:03 AM To: [EMAIL PROTECTED] Subject: Shared facilities (was Re: your mail) On Wed, 21 Aug 2002, David Lesher wrote: Unnamed Administration sources reported that N. Richard Solis said: If you haven't worked in an environment where you had to turn in your cellphone and pager at the front desk, show a badge to a camera around every corner, and get your office keys from a vending machine you dont know what real security looks like. You missed the places w/ real security. That's where the very polite Marine Security Guard with the 870 shotgun asks to see your badge again... Sigh, and in places with real security you rarely find enemies/competitors sitting in the same room. Exchange points are like the United Nations, not high security military bases. AMS-IX, Equinix, Linx/Telehouse, PAIX, etc provide a neutral facility for competitors to exchange network traffic. The facility operators provide a reasonable level of security, and try to keep the diplomats from punching each other. Its in all (most?) the competitors' self-interest to follow the rules. Let's not lose sight of the purpose of colocation/exchange points. If we start requiring you to be a US citizen and have top secret clearance in order to enter a colocation facility, we've probably decreased the usefulness of the exchange points.
Re: your mail
Sounds like a nuclear power plant I used to work at. Except the nuke plants don't trust the marines to do the job. They hire and train their own security teams. I had to go through more screening to work there than anything I've gone through re security clearances and the government. The scary thing is, (IMHO) the nuclear industry is being held up as the model for all other industries re security. Of course, there isn't the issue of many companies sharing one facility, which makes things far more interesting. A colo is no place for guns, imho. Jane David Lesher wrote: Unnamed Administration sources reported that N. Richard Solis said: If you haven't worked in an environment where you had to turn in your cellphone and pager at the front desk, show a badge to a camera around every corner, and get your office keys from a vending machine you dont know what real security looks like. You missed the places w/ real security. That's where the very polite Marine Security Guard with the 870 shotgun asks to see your badge again... -- A host is a host from coast to [EMAIL PROTECTED] no one will talk to a host that's close[v].(301) 56-LINUX Unless the host (that isn't close).pob 1433 is busy, hung or dead20915-1433
Re: your mail
On Tue, 20 Aug 2002, Ali Jackson wrote: Does any one else out there think smart hands at Equinix is a rip off? I can send a package over night to the IBX for less than what it costs to move it from the mailroom to my cage. Just curious Uh, yes. Equinix is a rip off in general. I got kicked out of Chicago using the side door. I was sick of the stupid man trap crap and noticed they had a door that was propped open in the back that leads outside. It was much easier to back the truck up there and go in and out. The whole thing is a joke, they spent a lot of cash to look good, but there is very little substance. Nathan Stratton nathan at robotics.net http://www.robotics.net
Re: your mail
Yes. Equinix security, while it looks very tough, is very easy to social engineer. Too much fluff, need more stuff. On Tue, 20 Aug 2002, Nathan Stratton wrote: On Tue, 20 Aug 2002, Ali Jackson wrote: Does any one else out there think smart hands at Equinix is a rip off? I can send a package over night to the IBX for less than what it costs to move it from the mailroom to my cage. Just curious Uh, yes. Equinix is a rip off in general. I got kicked out of Chicago using the side door. I was sick of the stupid man trap crap and noticed they had a door that was propped open in the back that leads outside. It was much easier to back the truck up there and go in and out. The whole thing is a joke, they spent a lot of cash to look good, but there is very little substance. Nathan Stratton nathan at robotics.net http://www.robotics.net -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
RE: your mail
I think that getting caught is a good indication that they take the security of the facility seriously. Some places will ban you forever if you violate their policies. The mantrap thing is there for a reason. People are always free to build out their own spaces however they wish. If you don't like their policies, don't colo there. Build your own. I like their approach of controlling access very tightly. Overkill is definitely better than underkill. My experience is that a lot of security measures that appear ridiculous or redundant actually act as a defense-in-depth strategy. Their practice of requiring a guard to leave the control booth to allow someone in instead of using a buzzer may seem stupid but serves an important but not entirely well-publicized purpose. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Paul Vixie Sent: Tuesday, August 20, 2002 2:54 PM To: [EMAIL PROTECTED] Subject: Re: your mail [EMAIL PROTECTED] (Nathan Stratton) writes: Uh, yes. Equinix is a rip off in general. I got kicked out of Chicago using the side door. I was sick of the stupid man trap crap and noticed they had a door that was propped open in the back that leads outside. It was much easier to back the truck up there and go in and out. The whole thing is a joke, they spent a lot of cash to look good, but there is very little substance. nevertheless PAIX hasn't made it to chicago yet, and equinix is quite a bit more neutral than a normal abovenet/exodus/att/qwest/ibm/uunet hosting center would be, and that makes them the only game in that town. i recommend that you work hard at helping them fix whatever it is they're doing wrong. think of your work in that regard as a public service. -- Paul Vixie
Re: your mail
There is no perfect location. Any common location has a certain level of insecurity. Im sure u could sneak in a squeeze bottle and spray equipment also. The point is, it is a relatively secure location, short of building your own facility or blding and manning it. Even many military installations are open to social engineering. Paul is absolutely right, as a good engineer and customer, put your suggestions in the suggestion box. Or access one of their people on the list. I think Bill Norton is easy to reach by email and so is their CTO. Dave At 18:54 + 8/20/02, Paul Vixie wrote: [EMAIL PROTECTED] (Nathan Stratton) writes: Uh, yes. Equinix is a rip off in general. I got kicked out of Chicago using the side door. I was sick of the stupid man trap crap and noticed they had a door that was propped open in the back that leads outside. It was much easier to back the truck up there and go in and out. The whole thing is a joke, they spent a lot of cash to look good, but there is very little substance. nevertheless PAIX hasn't made it to chicago yet, and equinix is quite a bit more neutral than a normal abovenet/exodus/att/qwest/ibm/uunet hosting center would be, and that makes them the only game in that town. i recommend that you work hard at helping them fix whatever it is they're doing wrong. think of your work in that regard as a public service. -- Paul Vixie -- David Diaz [EMAIL PROTECTED] [Email] [EMAIL PROTECTED] [Pager] Smotons (Smart Photons) trump dumb photons
RE: your mail
Equinix has show considerable interest in catering to the carrier market, and has always been very customer service oriented. Their security is generally good, and their security managers take the sort of stuff you are talking about very seriously. I have no doubt that they would take some serious action if told about a propped door. Their technical folks (Louie, Lane, etc) are sharp, and their helping hands is far above the level found at most carrier colos. In addition, they have folks like Bill Norton and Jay Adelson, folks with real service provider experience, who provide perspective to their ops folks, and who actively promote things that are good for the internet community like peering. Their Gigabit Peering Forums are at least as useful as NANOGs, sometimes quite a bit better. If you are looking for more basic, non-carrier neutral colo, it's out there - it might even be cheaper, in the very short run. However, getting lots of space in, say, Worldcom colos, may sound like a good deal, but it can cost you dearly in the long run, with incompetent or non-existing remote hands, dealing with very bad customer service, or bad security. - Daniel Golding Paul Vixie wrote [EMAIL PROTECTED] (Nathan Stratton) writes: Uh, yes. Equinix is a rip off in general. I got kicked out of Chicago using the side door. I was sick of the stupid man trap crap and noticed they had a door that was propped open in the back that leads outside. It was much easier to back the truck up there and go in and out. The whole thing is a joke, they spent a lot of cash to look good, but there is very little substance. nevertheless PAIX hasn't made it to chicago yet, and equinix is quite a bit more neutral than a normal abovenet/exodus/att/qwest/ibm/uunet hosting center would be, and that makes them the only game in that town. i recommend that you work hard at helping them fix whatever it is they're doing wrong. think of your work in that regard as a public service. -- Paul Vixie
Re: your mail
On Tue, Aug 20, 2002 at 03:08:22PM -0400, N. Richard Solis wrote: I think that getting caught is a good indication that they take the security of the facility seriously. Which is clearly exhibited by them leaving a side door propped open, or not checking or securing this door earlier --msa
Re: your mail
On Tue, Aug 20, 2002 at 02:07:49PM -0400, Nathan Stratton wrote: On Tue, 20 Aug 2002, Ali Jackson wrote: Does any one else out there think smart hands at Equinix is a rip off? I can send a package over night to the IBX for less than what it costs to move it from the mailroom to my cage. Just curious Uh, yes. Equinix is a rip off in general. I got kicked out of Chicago using the side door. I was sick of the stupid man trap crap and noticed they had a door that was propped open in the back that leads outside. It was much easier to back the truck up there and go in and out. The whole thing is a joke, they spent a lot of cash to look good, but there is very little substance. Did you try just asking if you could pull up to the loading dock? I can only speak to SJC and IAD, but since Equinix in Chicago is in the ghetto (how many blocks is it from the projects? :P) you would think it would have decent physical security. Yes if you get creative enough you can start talking about fake IDs and drugging someone and taking molds of their hands, but compared to the joke of most colo security it covers the areas where you could reasonably expect to see attacks. Personally I'm more concerned about quick and hassle free access than having to deal with a guard following me around the entire time. But I'm sure none of this matters to you, because you probably couldn't fight your urge to test the security, then got upset when they booted you for it, am I right? As for being a rip off, I suggest you price other carrier neutral colo and then come back with that. And no I don't have any vested interest in Equinix, I've even had my own bad experiences with their security (which were delt with promptly). But they're still reasonable to deal with, offer an all around excellent service, and they've done a much better job on security and other fronts than other colos. As for the original poster, remote hands service is expensive, and smart hands is usually for smart services. If all you want is hands either drag your...self down to the colo and start lifting, or hire your own $10/hr rack and stack monkeys. At any rate, this has no place on nanog. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
RE: your mail
Does any one else out there think smart hands at Equinix is a rip off? I can send a package over night to the IBX for less than what it costs to move it from the mailroom to my cage. Just curious Uh, yes. Equinix is a rip off in general. I got kicked out of Chicago using the side door. I was sick of the stupid man trap crap and noticed they had a door that was propped open in the back that leads outside. It was much easier to back the truck up there and go in and out. The whole thing is a joke, they spent a lot of cash to look good, but there is very little substance. --- There are a few companies that spent money on bullet-resistant reception areas [the purpose has always eluded me] -- when you ask if the loading dock and all other entrances are similarly reinforced, you get the no, of course not!. I guess the thought process is: basically assume the bad guys will go through the front door and cooperate with the mantrap. Then again (bringing this back to EQIX) -- they didn't spend money on expensive signage, because no one would guess where the door is -- right? So that's more of a savings than some other companies have/had/whatever. DJ I'm curious -- did they kick you out for the day, or terminate your contract and move you out?
RE: your mail
Leaving or forcing doors to be propped open generally triggers an alarm that prompts a visit from someone in security. It is entirely possible that someone who worked at the facility informed the security staff of what they were doing because they needed to leave the door open to fetch a package or something that was going to be moved through that door. It's also entirely possible that someone working there was violating the security policy entirely. That happens as well. I would need many more fingers and toes to count the number of sleeping guards I've caught at colo sites. The point is: people do dumb things that compromise security for everyone in order to make their own lives easier. A good security plan anticipates these lapses and puts measures in place to deal with them. If you haven't worked in an environment where you had to turn in your cellphone and pager at the front desk, show a badge to a camera around every corner, and get your office keys from a vending machine you dont know what real security looks like. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Majdi S. Abbas Sent: Tuesday, August 20, 2002 3:13 PM To: N. Richard Solis Cc: [EMAIL PROTECTED] Subject: Re: your mail On Tue, Aug 20, 2002 at 03:08:22PM -0400, N. Richard Solis wrote: I think that getting caught is a good indication that they take the security of the facility seriously. Which is clearly exhibited by them leaving a side door propped open, or not checking or securing this door earlier --msa
RE: your mail
On Tue, 20 Aug 2002, N. Richard Solis wrote: Leaving or forcing doors to be propped open generally triggers an alarm that prompts a visit from someone in security. It is entirely possible that someone who worked at the facility informed the security staff of what they were doing because they needed to leave the door open to fetch a package or something that was going to be moved through that door. It's also entirely possible that someone working there was violating the security policy entirely. That happens as well. I would need many more fingers and toes to count the number of sleeping guards I've caught at colo sites. Correct, I am sorry I think that is my point. There are a lot of things that they SHOULD have been doing, but they were not. I am saying they spent lots of money on a security image and not on security. They never found me using the door and that is a problem, when I let them know about their issues they rather shut me up then deal with them. The point is: people do dumb things that compromise security for everyone in order to make their own lives easier. A good security plan anticipates these lapses and puts measures in place to deal with them. If you haven't worked in an environment where you had to turn in your cellphone and pager at the front desk, show a badge to a camera around every corner, and get your office keys from a vending machine you dont know what real security looks like. I know what real security looks like, I also know what real security is. I am saying that I am willing to pay for real security, but I am not willing to page for the image of real security and go through the hassle of the image of real security when there is no real security. I don't know about all of their sights, but at least two have the security image when you walk in, but the rest of the building and other entrances have less then my house. Nathan Stratton nathan at robotics.net http://www.robotics.net
Re: your mail
Speakig of paix's and locations, I know the mfn filings have held up progress but I wondered and maybe others on this list wonder what the status of the paix nyiix interconnection might be? On 20 Aug 2002, Paul Vixie wrote: [EMAIL PROTECTED] (Nathan Stratton) writes: Uh, yes. Equinix is a rip off in general. I got kicked out of Chicago using the side door. I was sick of the stupid man trap crap and noticed they had a door that was propped open in the back that leads outside. It was much easier to back the truck up there and go in and out. The whole thing is a joke, they spent a lot of cash to look good, but there is very little substance. nevertheless PAIX hasn't made it to chicago yet, and equinix is quite a bit more neutral than a normal abovenet/exodus/att/qwest/ibm/uunet hosting center would be, and that makes them the only game in that town. i recommend that you work hard at helping them fix whatever it is they're doing wrong. think of your work in that regard as a public service.
Re: your mail
Speakig of paix's and locations, I know the mfn filings have held up progress but I wondered and maybe others on this list wonder what the status of the paix nyiix interconnection might be? until mfn finishes selling paix, there will likely be no progress on this.
RE: your mail
Then the appropriate person to talk to is the account manager. Catching a problem yourself doesn't do anyone any good if the management of the facility (or the company) isn't involved. My experience is that a LOT of companies want to hear from customers when things go amiss. They can't always rely on their own employees to let them know when the are falling down on the job. I've gotten corrective action form people just by threatening to bring in a higher management layer. People would rather fix a problem themselves than allow their management to fix it for them. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Nathan Stratton Sent: Tuesday, August 20, 2002 5:07 PM To: N. Richard Solis Cc: Majdi S. Abbas; [EMAIL PROTECTED] Subject: RE: your mail On Tue, 20 Aug 2002, N. Richard Solis wrote: Leaving or forcing doors to be propped open generally triggers an alarm that prompts a visit from someone in security. It is entirely possible that someone who worked at the facility informed the security staff of what they were doing because they needed to leave the door open to fetch a package or something that was going to be moved through that door. It's also entirely possible that someone working there was violating the security policy entirely. That happens as well. I would need many more fingers and toes to count the number of sleeping guards I've caught at colo sites. Correct, I am sorry I think that is my point. There are a lot of things that they SHOULD have been doing, but they were not. I am saying they spent lots of money on a security image and not on security. They never found me using the door and that is a problem, when I let them know about their issues they rather shut me up then deal with them. The point is: people do dumb things that compromise security for everyone in order to make their own lives easier. A good security plan anticipates these lapses and puts measures in place to deal with them. If you haven't worked in an environment where you had to turn in your cellphone and pager at the front desk, show a badge to a camera around every corner, and get your office keys from a vending machine you dont know what real security looks like. I know what real security looks like, I also know what real security is. I am saying that I am willing to pay for real security, but I am not willing to page for the image of real security and go through the hassle of the image of real security when there is no real security. I don't know about all of their sights, but at least two have the security image when you walk in, but the rest of the building and other entrances have less then my house. Nathan Stratton nathan at robotics.net http://www.robotics.net
RE: your mail
On Tue, 20 Aug 2002, N. Richard Solis wrote: Then the appropriate person to talk to is the account manager. Catching a problem yourself doesn't do anyone any good if the management of the facility (or the company) isn't involved. That presumes there is a single account manager. With Equinix, there are no less than 5 different people I need to call depending on what I need. They've shifted account management costs back on the customer. /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ Patrick Greenwell Asking the wrong questions is the leading cause of wrong answers \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
RE: your mail
On Tue, 20 Aug 2002, Deepak Jain wrote: I'm curious -- did they kick you out for the day, or terminate your contract and move you out? Basically they said they would ban me personally if I gave there security people a hard time about their security. I don't think they ever would terminate a contract if you were paying their sick rates. Nathan Stratton nathan at robotics.net http://www.robotics.net
RE: your mail
On Tue, 20 Aug 2002, Deepak Jain wrote: I'm curious -- did they kick you out for the day, or terminate your contract and move you out? Basically they said they would ban me personally if I gave there security people a hard time about their security. I don't think they ever would terminate a contract if you were paying their sick rates. Good contract point here - if for any reason Customer's key personnel are not able to access the facility or equipment, at Customer's option Agreement may be terminated with 30 days notice. That will make everyone a little more polite, IMO. Deepak Jain AiNET
Re: Hovercraft for deliveries to the fifth floor loading dock (was Re: Your mail)
I am not an ex-employee of Equinix, so here's my 2 cents: When we built the IBXs, having spent a couple of years listening to you folks tell me what you want at the PAIX and elsewhere, I basically learned it was impossible to satisfy everyone. If you please one network engineer, you're going to annoy another one, and that's just the way it works. In the immortal words of Stephen Stuart, Sorry. Apparently our secret plan to shut up Nathan did fail miserably. ;) We'll have to set the hand geometry readers to electrocute him on his next appearance at the IBX. 1) Fire codes and other local ordinances interfered with my grand plan to bury you in concrete and eliminate fire exits. ;) In other words, we have no choice put to put fire exits in there, otherwise many of you would die in a fire due to the sheer size of our facilities. Fire doors don't work very well if you can't open them. In some regions, we're allowed by code to lock it shut for a delay and theoretically that's enough time to send a guard to hunt you down and remove you. In others we need to let it open, but an alarm goes off (sometimes silently, other times very loudly) to accomplish the same effect. If Nathan propped open a door and was able to enter/exit without being caught then that was a failure and one I'd like to address... In any case, yes we do have a camera watching you, and we do keep records of all that, so if you think it's a big security hole and plan on balancing that GSR on one toe into the back of your pickup so you can sell it on the street corner go ahead and try. Don't be surprised if I don't write you in jail. 2) Customers are given one point of contact they can call for anything. You know, it's that game... if you do what one person wants it annoys another... So therefore, just like engineers love to call their favorite go-getter of the day, it's ok for customers to call account reps, SEs, or even network engineers and folks like me. We don't care. However, if you want to call the ERC we figure that's fine as well. We thought everyone would want to bypass humans all together and use a web site. We were proven wrong on that front, though some of the more organized customers use the web interface regardless. So you don't HAVE to call five different people, but hell, if you want to, have a field day. (What? You mean there is flexibility? Preposterous!!!) Finally, remember the point of all this... peering points didn't take into account the physical issues associated with colo, and we tried to address them from the network engineer's perspective...paying special attention to the VERY different colocation needs for different customers. Oh yeah, and then try and duplicate it exactly in seven buildings. At 3am one day maybe even Nathan can appreciate the way we designed them... Being a colo provider is a necessary evil needed to accomplish the much more important goals of solving certain other exchange point issues. On Tue, Aug 20, 2002 at 06:50:00PM -0400, Sean Donelan wrote: On Tue, 20 Aug 2002, Nathan Stratton wrote: Correct, I am sorry I think that is my point. There are a lot of things that they SHOULD have been doing, but they were not. I am saying they spent lots of money on a security image and not on security. They never found me using the door and that is a problem, when I let them know about their issues they rather shut me up then deal with them. Obviously their secret plan to shut you up failed :-) Like commercial ventures, there is a certain amount of fluff and puffery. Banks still get robbed even with that really, really thick door on the vault. Most car commercials have fine print at the bottom saying don't do this insane thing. It gives the sales people something to talk about. Stick your fingers in your ears and ignore the sales person until you want to talk about discounts. Any technically savvy person should be able to do due dilegence and determine if a facility meets his needs. The question isn't really about security, but how it compares to other facilities of a similar caliber. You could drive a tank, but its really hard to park and gets lousy gas milage. Comparing a car to a tank isn't very useful. Comparing a Volvo to a Saab might provide information to make an informed choice. Is Equinix (PAIX, MFN, NOTA, etc) less secure than NORAD? Yes. Are there things I wish they did differently? Yes. Have they ever left a door unlocked? Yes. Have they ever made a mistake? Yes. Is Equinix a clean, secure, well-run facility I would trust to house my equipment? Yes. Would I also buy insurance and consider a diverse, back up site for my equipment? Yes. Disclosure: I'm an ex-employee of Equinix. -- [ Jay Adelson [EMAIL PROTECTED] ] [ Founder, Chief Technology Officer Work: +1-650-316-6000 ] [ Equinix, Inc., Mountain View, CA Fax: +1-650-316-6904 ]
Re: Hovercraft for deliveries to the fifth floor loading dock (wasRe: Your mail)
On Tue, 20 Aug 2002, Jay Adelson wrote: 2) Customers are given one point of contact they can call for anything. I'm your customer and I'm telling you that I haven't been and when I've specifically asked for a single point of contact I've been told that I need to contact a variety of people based on what it is I need. You know, it's that game... if you do what one person wants it annoys another... So therefore, just like engineers love to call their favorite go-getter of the day, it's ok for customers to call account reps, SEs, or even network engineers and folks like me. We don't care. Really? --- From: Christina Canady [EMAIL PROTECTED] CC: Duane MacKenzie [EMAIL PROTECTED] Subject: RE: Packing Slips Date: Fri, 2 Aug 2002 14:43:56 -0700 I believe Duane has responded regarding the packing slips. On another note, all correspondence to the IBX needs to go through me or the ERC in the future. We ask all our customers not to call or email the IBXes directly. Thank you, CC -- I invite you to take any further correspondence regarding this issue private, and look very forward to your response. /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ Patrick Greenwell Asking the wrong questions is the leading cause of wrong answers \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
Re: Hovercraft for deliveries to the fifth floor loading dock (was Re: Your mail)
Patrick, Yes, really! That's what the ERC is for. I guess the confusion is outside your email thread, which indicates as such... But yes, the single point is supposed to be the ERC. Feel free to contact me with specifics... -Jay On Tue, Aug 20, 2002 at 05:29:43PM -0700, Patrick wrote: On Tue, 20 Aug 2002, Jay Adelson wrote: 2) Customers are given one point of contact they can call for anything. I'm your customer and I'm telling you that I haven't been and when I've specifically asked for a single point of contact I've been told that I need to contact a variety of people based on what it is I need. You know, it's that game... if you do what one person wants it annoys another... So therefore, just like engineers love to call their favorite go-getter of the day, it's ok for customers to call account reps, SEs, or even network engineers and folks like me. We don't care. Really? --- From: Christina Canady [EMAIL PROTECTED] CC: Duane MacKenzie [EMAIL PROTECTED] Subject: RE: Packing Slips Date: Fri, 2 Aug 2002 14:43:56 -0700 I believe Duane has responded regarding the packing slips. On another note, all correspondence to the IBX needs to go through me or the ERC in the future. We ask all our customers not to call or email the IBXes directly. Thank you, CC -- I invite you to take any further correspondence regarding this issue private, and look very forward to your response. /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ Patrick Greenwell Asking the wrong questions is the leading cause of wrong answers \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ -- [ Jay Adelson [EMAIL PROTECTED] ] [ Founder, Chief Technology Officer Work: +1-650-316-6000 ] [ Equinix, Inc., Mountain View, CA Fax: +1-650-316-6904 ]
