Re: Abuse response [Was: RE: Yahoo Mail Update]
William Herrin wrote: On Tue, Apr 15, 2008 at 8:49 PM, Martin Hannigan [EMAIL PROTECTED] wrote: Abuse desk is a $0 revenue operation. Is it not obvious what the issue is? Martin, So is marketing, yet marketing does have an impact on revenue. It can be useful to explain the abuse desk as being just another form of marketing, another form of reputation management that happens to be specific to Internet companies. Handling the abuse desk well (or poorly) builds (or damages) the brand. Even IF the reputation of an abuse desk had any effect at all on bringing in revenue (doubtful) ... I'm quite certain that dollar for dollar, the ROI on investment in Marketing generates MUCH greater revenue returns than investment in Abuse desk staff. Properly staffing an abuse desk is something a business does because It Is The Right Thing To Do, not because it's the best investment for their marketing dollars. jc
RE: Abuse response [Was: RE: Yahoo Mail Update]
So how do the little guys play in this sandbox? 3rd-party aggregation. Where do RBLs get there data? They act as a 3rd party to aggregate data from many others. - It needs to be simple to use. Web forms are a non-starter. If you have the ability to accept reports via an HTTP REST application, it wouldn't hurt to put up a web form so that people can try it out. - The output from any parsers needs to be human readable. ARF is the only thing that meets this requirement http://mipassoc.org/arf/ However, you should consider accepting input as IODEF as well. Just use ARF for the ouput that you submit to the abuse desks. - I'd like to see an actual response beyond an autoreply saying that you can't tell me who the customer is or what actions were taken. Now you are asking the abuse desks to modify their software and processes to meet your needs. I can't see them ever providing a response per report, however if enough people buy into a standard reporting system, like ARF, then you might get ISPs to accept some kind of report-origin code and then allow you to periodically request resolution reports for all reports coming from that report-origin. - I like dealing with other small operations and edus because humans actually do read the reports, and things get done (Thanks!). If people had succeeded in cleaning up the abuse problems in 1995 when the human touch was still feasible, we would not have the situation that we have today. Automation is the only way to address the flood of abuse email, the huge number of people originating abuse, and the agile tactics of the abusers. You just have to accept that people will not read your reports, and will not act on your reports. What they will do is feed your reports into automated systems that use AI techniques to define tasks for the abuse desk to act upon. Consider this. Any single point source of abuse, say a single broadband PC in a botnet, will spew out spam or DDOS to hundreds of destinations. If 20 of these destinations submit ARF reports, and you are one of these 20, then there is a 5% chance that your report has anything wort acting upon. 95% of the time, you will be reporting something that the abuse desk has already acted upon and it would be a waste of abuse desk resources to read and reply to your report. On the other hand, it can be very useful for the automated system to process your report for statistical purposes and to provide a better understanding of how that particular botnet functions. I've given up sending abuse reports to large consumer ISPs and all freemail providers because I'm not a member of the club. Any response that I'm lucky enough to get generally says something like You did not include the email headers in your complaint so we are closing this incident when I reported and FTP brute force. This is why we need *MORE* automation between providers. Then there is less room for human error in wading through a mass of reports trying to pick out the ones which can be fixed. --Michael Dillon
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 08:49:39PM -0400, Martin Hannigan wrote: Abuse desk is a $0 revenue operation. Is it not obvious what the issue is? Two points, the first of which is addressed to this and the second of which is more of a recommended attitude. 1. There is no doubt that many operations consider it so, but it's really not. Operations which don't adequately deal with abuse issues are going to incur tangible and intangible costs (e.g., money spent cleaning up local messes and getting off numerous blacklists, loss of business due to reputation, etc.). Those costs are likely to increase as more and more people become increasingly annoyed with abuse-source operations and express that via software and business decisions. I'll concede that this is really difficult to measure (at the moment) but it's not zero. 2. When one's network operation abuses someone (or someone else's operation), you owe them a fix, an explanation, and an apology. After all, it happened in your operation on your watch, therefore you're personally responsible for it. And when someone in that position -- a victim of abuse -- has magnanimously documented the incident and reported it to you, thus providing you with free consulting services -- you owe them your thanks. After all, they caught something that got by you -- and they've shared that with you, thus enabling you to run a better operation, which in turn means fewer future abuse incidents, which in turn means lower tangible and intangible costs. And far more importantly, it means being a better network neighbor, something we should all be working toward all the time. ---Rsk
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Wed, Apr 16, 2008 at 11:07:42AM +0100, [EMAIL PROTECTED] wrote: If people had succeeded in cleaning up the abuse problems in 1995 when the human touch was still feasible, we would not have the situation that we have today. Automation is the only way to address the flood of abuse email, the huge number of people originating abuse, and the agile tactics of the abusers. I agree with this and with pretty much everything else you wrote. But... If an operation is permitting itself to be such a systemic, persistent source of abuse that the number of abuse reports it's receiving (which everyone knows is tiny fraction of the number it *could* be receiving) requires automation...isn't that a pretty good sign that whatever's being done to control abuse isn't working? The solution to that isn't to put in place higher levels of automation: the solution to to that is to *solve the underlying problems* so that higher levels of automation aren't necessary. ---Rsk
RE: Abuse response [Was: RE: Yahoo Mail Update]
So who's the third-party for the little guy that aggregates abuse reports? I know we consume Spamcop reports which works very well for us. I'm not sure who feeds them data. Ideally I would like to be able to submit data to them in an automated fashion, but the spam appliance I have doesn't have that checkbox. If the abuse desk has already acted upon it, why not have the automated system let me know? Frank -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Wednesday, April 16, 2008 5:08 AM To: nanog@merit.edu Subject: RE: Abuse response [Was: RE: Yahoo Mail Update] So how do the little guys play in this sandbox? 3rd-party aggregation. Where do RBLs get there data? They act as a 3rd party to aggregate data from many others. snip Consider this. Any single point source of abuse, say a single broadband PC in a botnet, will spew out spam or DDOS to hundreds of destinations. If 20 of these destinations submit ARF reports, and you are one of these 20, then there is a 5% chance that your report has anything wort acting upon. 95% of the time, you will be reporting something that the abuse desk has already acted upon and it would be a waste of abuse desk resources to read and reply to your report. On the other hand, it can be very useful for the automated system to process your report for statistical purposes and to provide a better understanding of how that particular botnet functions. snip --Michael Dillon
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Wed, 16 Apr 2008 00:38:33 CDT, Chris Boyd said: - I'd like to see an actual response beyond an autoreply saying that you can't tell me who the customer is or what actions were taken. Well, let's see. If you're reporting abuse coming from my AS, it's almost certainly one of 2 things: 1) Some poor soul got zombied in a drive-by fruiting and was part of a botnet. At this point, it doesn't really matter *who* the customer was, because he was essentially a Joe Sixpack. Action taken is almost certainly some variant on he's been told to disinfect the machine before getting back on the net. So it's unclear what, if anything, you want us to do, except possibly send you a canned We found the machine and dealt with it after the fact. 2) Somebody decided to intentionally do something naughty. At that point, it's a very good likelyhood that we *can't* tell you who it was, because there may be some combination of litigation and prosecution (and in our case, most likely some internal judicial action) so there's a whole swarm of privacy laws and we don't comment on ongoing investigations/litigations policy. And since these things can drag on for weeks or months, there may not be any final resolution for quite some time, so all you'll get back is a We found the problem and it will eventually be disposed of... Basically, 99.8% of the time, no response other than We found it and dealt with it is actually suitable, and the other 0.2% of the time, you're about to get dragged into an ongoing investigation, so expect a Hold Evidence order on your fax in a few minutes.. ;) So what sort of response did you actually *want*? pgpwl7fz8B5YY.pgp Description: PGP signature
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 8:49 PM, Martin Hannigan [EMAIL PROTECTED] wrote: Abuse desk is a $0 revenue operation. Is it not obvious what the issue is? Martin, So is marketing, yet marketing does have an impact on revenue. It can be useful to explain the abuse desk as being just another form of marketing, another form of reputation management that happens to be specific to Internet companies. Handling the abuse desk well (or poorly) builds (or damages) the brand. Regards, Bill Herrin -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Abuse response [Was: RE: Yahoo Mail Update]
It can be useful to explain the abuse desk as being just another form of marketing, another form of reputation management that happens to be specific to Internet companies. Is it? I mean, I may know that (a hypothetical) example.com is a pink-contract-signing batch of incompetents who spew spam like a bulemic firehose. You may know that. 10,000 other mail administrators may know that. But once they have signed up 2.3 million users with example.com they are too big (for most email administrators) to block, so at that point the cost of disbanding their abuse desk and pointing complaints to /dev/null is nil. Handling the abuse desk well (or poorly) builds (or damages) the brand. ...among people who are educated among such things. Unfortunately, people with clue are orders of magnitude short of a majority, and the rest of the world (ie: potential customers) wouldn't know an abuse desk from a self-abuse desk. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Wednesday 16 April 2008 17:47, Dave Pooser wrote: It can be useful to explain the abuse desk as being just another form of marketing, another form of reputation management that happens to be specific to Internet companies. Is it? .. SNIP good points about abuse desks .. In the specific case that started this (Yahoo), then I think there is a marketing issue. Ask anyone in the business if I want a free email account who do I use.. and you'll get the almost universal answer Gmail. Mostly this is because Hotmail delete email randomly, Yahoo struggle with the volumes, and everyone forgets AOL do free accounts (although it is painfully slow and the documentation is incomplete). But it is in part that Google do actually answer enquiries still, be they abuse or support. Yahoo occassionally manage an answer, usually not to the question you asked, or asking for information already supplied. AOL - well you can get an answer from their employee who watches Spam-L, but directly not a chance. So it is a competitive market, and the opinion of those in the know matters (a little -- we could make more noise!). Although the tough one to compete with is Hotmail, since their computer offers it to them every time they reinstall, and those reinstalling more often have least clue, but eventually realise having their email on THEIR(!) PC is a bad idea. But yes, abuse desk is only a minor issue in that market, but if you don't deal with abuse, it will cost the bottom line for email providers. I think for people mostly providing bandwidth, email is still largely irrelevant, even at the hugely inflated levels the spammers cause it is still a minor %age, favicons (missing or otherwise) probably cause nearly as much traffic.
Re: Abuse response [Was: RE: Yahoo Mail Update]
Dave Pooser wrote: Handling the abuse desk well (or poorly) builds (or damages) the brand. ...among people who are educated among such things. Unfortunately, people with clue are orders of magnitude short of a majority, and the rest of the world (ie: potential customers) wouldn't know an abuse desk from a self-abuse desk. I think that depends on the nature of the abuse desk, how it interfaces with other networks and the customer base. Of course, I get to be the NOC guy and the abuse guy here. It's nice to have less than a million customers. However, I find that how NOC issues and abuse issues are handled are very similar. It is, of course, easier to reach another NOC than it is the senior abuse staff that actually have clue, generally. Both departments need a certain amount of front line protection to keep them from being swamped with issues that can be handled by others. Never the less, when they can interface with customers and with the other departments that spend more time with customers, it does improve the company's service level. If there is a routing, firewalling, or email delivery issue with a much larger network, the effectiveness of the NOC/Abuse Dept will determine how well the customers will handle the interruption. If the company has built trust with the customer and related to them in a personal way, then the customer will in turn tend to be more understanding of the issues involved, or in some cases at least point their anger at the right company. -Jack Learning to mitigate the damage caused by Murphy's law.
Re: Abuse response [Was: RE: Yahoo Mail Update]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- [EMAIL PROTECTED] wrote: So what sort of response did you actually *want*? Actually, I'm more concerned with alerting you that someone inserted a nasty .js or iFrame on one of your websites and I'd like to you to clean it up, thanks. ;-) I'm not so concerned about alerting you to botted student computers... that's another issue entirely. :-) - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIBj/nq1pz9mNUZTMRAmlKAJ4v/KIvHlKvO1MDF97Ed1T9RkpnjgCgvvRC CLUNjfK4mZcQOga42UgY9og= =7OPB -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: Abuse response [Was: RE: Yahoo Mail Update]
On 16 Apr 2008, at 13:33 , Simon Waters wrote: Ask anyone in the business if I want a free email account who do I use.. and you'll get the almost universal answer Gmail. I think amongst those not in the business there are regional trends, however. Around this neck of the woods (for some reason) the answer amongst your average, common-or-garden man in the street is yahoo!. I don't know why this is. But that's my observation. There are also the large number of people using Y! mail who don't realise they're using Y! mail, because the telco or cableco they use for access have outsourced mail operations to Y!, and there are still (apparently) many people who assume that access providers and mail providers should match. In those cases choice of mail provider may have far more to do with price of tv channel selections or availability of long-distance voice plans than anything to do with e- mail. So, with respect to your other comments, correlation between technical/ operational competence and customer choice seems weak, from my perspective. If there's competition, it may not driven by service quality, and the conclusion that well-staffed abuse desks promote subscriber growth is, I think, faulty. Joe
Re: Abuse response [Was: RE: Yahoo Mail Update]
Subject: Re: Abuse response [Was: RE: Yahoo Mail Update]
From: [EMAIL PROTECTED]
Date: Wed, 16 Apr 2008 12:02:02 -0400
On Wed, 16 Apr 2008 00:38:33 CDT, Chris Boyd said:
- I'd like to see an actual response beyond an autoreply saying that you
can't tell me who the customer is or what actions were taken.
Well, let's see. If you're reporting abuse coming from my AS, it's almost
certainly one of 2 things:
[[ sneckcausations ]]
Basically, 99.8% of the time, no response other than We found it and dealt
with it is actually suitable, and the other 0.2% of the time, you're about
to get dragged into an ongoing investigation, so expect a Hold Evidence
order on your fax in a few minutes.. ;)
So what sort of response did you actually *want*?
Speaking strictly for myself, the wish-list for an ack is (not necessarily in
priority order):
1) appreciation for my contributed time/effort in helping them keep _their_
network clean.
2) an ack that they _have_found_ the source. I generally don't care 'who'
it was, just that they *have* been found, and STOPPED.
3) an indication that the immediate issue has been fixed, and that steps
have been taken to prevent future recurrance.Again, the actual
'details' of what has been done are relatively unimportant.
4) *WHEN* the 'fix' was implemented. Then I know if I see 'more of the
same _before_ that time, I don't need to report it, =AND= if I see
stuff occuring _after_ that time, that it is a 'new and different'
problem that _does_ need to be reported.
This is more about _how_ you say things, than the details of what you actually
say.
Replies -- _days_ later -- along the lines of thanks for the report, due to
volume of complaints we won't be able to tell you anything about what we find,
or do cause much grinding of teeth.
Replies that say: This appears to be the same as something that has already
been reported to us by others. We have looked into things, confirmed it was
happening, and put a stop to it as of {timestamp}. If you see any more of this
activity from that source _after_ that time please email us immediately with
the string {token} in the subject line. _do_ give the originater 'warm
fuzzies', and can be more-or-less trivially generated by a good trouble-
ticket system. Especially with reasonable front-end automation for recognizing
'duplicate' complaints.
At the good end, I've gotten replies saying: the customer has been contacted,
and they immediately took the affected machine off-line for sterilization;
even we have been unable to contact the customer, and have pulled their
circuit until they *do* contact us.
Note: that last message was received about 4 hours after sending the problem
notice, and about 2 hours after what would have been the normal 'start of
business' in the locale of the problem. That provider wears a *BIG* white
hat in my books. Not so much for telling me what they did, but for the speed
of reaction.
Contrast those responses with a major national who doesn't send any responses
*and* has an admitted policy of giving customers _a_week_after_notification_
of having an infected machine on their network to get the machine off-line or
otherwise dealt with. And it can take _days_ to get the notification to the
customer. (they just send an email to the business contact -- notify them late
friday and the clock doesn't start running until Monday morning. *sigh*)
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Wed, Apr 16, 2008 at 03:39:05PM -0400, Joe Abley wrote: On 16 Apr 2008, at 13:33 , Simon Waters wrote: Ask anyone in the business if I want a free email account who do I use.. and you'll get the almost universal answer Gmail. I think amongst those not in the business there are regional trends, however. Around this neck of the woods (for some reason) the answer amongst your average, common-or-garden man in the street is yahoo!. I don't know why this is. But that's my observation. In my experience, Gmail tends to be the preferred freemail acount among geeks and techies. Y! mail and Hotmail are preferred by the (non-techie) man and woman on the street. I think this is largely due to branding. So, with respect to your other comments, correlation between technical/ operational competence and customer choice seems weak, from my perspective. If there's competition, it may not driven by service quality, and the conclusion that well-staffed abuse desks promote subscriber growth is, I think, faulty. Also, IME, the business community tends to perceive marketing as a profit center (whether or not it actually is), because they understand it and can measure the ROI they get from it. This may not be the case in companies with executives who came from the tech side, however, but it's still more common for executives to have more of a business than technical background. --gregbo
Re: Abuse response [Was: RE: Yahoo Mail Update]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Suresh Ramasubramanian [EMAIL PROTECTED] wrote: If you send reports with lots of legal boilerplate, or reports with long lectures on why you expect an INSTANT TAKEDOWN, and send them to a busy abuse queue, there is no way - and zero reason - for the ISP people to prioritize your complaint above all the other complaints coming in. Having elided the rest of this exchange, and also understanding exactly what you are talking about, I encourage you to elaborate on the point you are trying to make... As you well know, there are many of us who have been working on this particular issue for years, with wildly varying degrees of success. There is no pat answer... - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIBEFTq1pz9mNUZTMRArvBAJ0XvKGXrL5yCKttE/0g1cxpkuWwAwCcCnw8 7Y8Q1TPWRnpvVH/5fdh5r2c= =Gcoo -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 11:04 AM, Paul Ferguson [EMAIL PROTECTED] wrote: In fact, we have done just that -- develop a standard boilerplate very similar to what PIRT uses in its notification(s) to the stakeholders in phishing incidents. The boilerplate is no damned use. PIRT - and you - should be focusing on feedback loops, and that would practically guarantee instant takedown, especially when the notification is sent by trusted parties. Again, our success rate is somewhere in the 50% neighborhood. With the larger providers it will get to 100% once you go the feedback loop route. Do ARF, do IODEF etc. You will find it much easier for abuse desks that care to process your reports. You will also find it easier to feed these into nationwide incident response / alert systems like Australia's AISI (google it up, you will like the concept I think) srs
Re: Abuse response [Was: RE: Yahoo Mail Update]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Suresh Ramasubramanian [EMAIL PROTECTED] wrote: Do ARF, do IODEF etc. You will find it much easier for abuse desks that care to process your reports. You will also find it easier to feed these into nationwide incident response / alert systems like Australia's AISI (google it up, you will like the concept I think) Really. How many people are actually doing IODEF? http://www.terena.org/activities/tf-csirt/iodef/ Honestly? And the other regional formats? This is kind of what I mean when I talk about disjointed and discombobiulated processes of reporting abuse. It should be simple -- not require a freeking full-blown standard. - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIBEo/q1pz9mNUZTMRAvphAKCTmSmbRHBCq9wuK9U+PDR+PFxWtQCgpV8s z5EJEitF6mIhHspeNuVNMOU= =x2Qh -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: Abuse response [Was: RE: Yahoo Mail Update]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Suresh Ramasubramanian [EMAIL PROTECTED] wrote: Do ARF, do IODEF etc. You will find it much easier for abuse desks that care to process your reports. You will also find it easier to feed these into nationwide incident response / alert systems like Australia's AISI (google it up, you will like the concept I think) And further, looking at IODEF in particular, this is doomed: it requires more than two simple steps to report abuse. The proof is in the pudding. - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIBEuNq1pz9mNUZTMRAt94AJ9NYRFDM1UKMs5GEO9klDeLDWajdwCfaB7M NLS2W3SAD9fZiV1ScGthlPI= =+V6W -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 11:55 AM, Paul Ferguson [EMAIL PROTECTED] wrote: Really. How many people are actually doing IODEF? http://www.terena.org/activities/tf-csirt/iodef/ AISI - for example - and AISI feeds the top 25 australian ISPs - takes IODEF as an input And MAAWG does ARF, quite simple to use as well .. but they would take a standard format (with an RFC yet) if you and some other major players 1. Offer iodef (or say ARF) feeds 2. Tell them youre offering these feeds It should be simple -- not require a freeking full-blown standard. Its a standard. And it allows automated parsing of these complaints. And automation increases processing speeds by orders of magnitude.. you dont have to wait for an abuse desker to get to your email and pick it out of a queue with hundreds of other report emails, and several thousand pieces of spam [funny how [EMAIL PROTECTED] type addresses end up in so many spammer lists..] srs
Re: Abuse response [Was: RE: Yahoo Mail Update]
do you remember the days when some of us would only take routing table updates from andrew partan, because we trusted him? that's what it's like now wrt takedowns. do not minimize the use of malicious takedowns by twits and bad guys, who fabricate a report of misfeasance to get their enemies taken down. On Apr 15, 2008, at 7:47 AM, Paul Ferguson wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Suresh Ramasubramanian [EMAIL PROTECTED] wrote: If you send reports with lots of legal boilerplate, or reports with long lectures on why you expect an INSTANT TAKEDOWN, and send them to a busy abuse queue, there is no way - and zero reason - for the ISP people to prioritize your complaint above all the other complaints coming in. Having elided the rest of this exchange, and also understanding exactly what you are talking about, I encourage you to elaborate on the point you are trying to make... As you well know, there are many of us who have been working on this particular issue for years, with wildly varying degrees of success. There is no pat answer... - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIBEFTq1pz9mNUZTMRArvBAJ0XvKGXrL5yCKttE/0g1cxpkuWwAwCcCnw8 7Y8Q1TPWRnpvVH/5fdh5r2c= =Gcoo -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 12:31:33PM +0530, Suresh Ramasubramanian wrote: On Tue, Apr 15, 2008 at 11:55 AM, Paul Ferguson [EMAIL PROTECTED] wrote: [snip] It should be simple -- not require a freeking full-blown standard. Its a standard. And it allows automated parsing of these complaints. And automation increases processing speeds by orders of magnitude.. you dont have to wait for an abuse desker to get to your email and pick it out of a queue with hundreds of other report emails, and several thousand pieces of spam [funny how [EMAIL PROTECTED] type addresses end up in so many spammer lists..] It cannot be understated that even packet pushers and code grinders who care get stranded in companies where abuse handling is deemed by management to be a cost center that only saps resources. Paul, you are doing a serious disservice to those folks in specific, and working around such suit-induced damage in general, by dismissing any steps involving automation. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
Re: Abuse response [Was: RE: Yahoo Mail Update]
I largely concur with the points that Paul's making, and would like to augment them with these: - Automation is far less important than clue. Attempting to compensate for lack of a sufficient number of sufficiently-intelligent, experienced, diligent staff with automation is a known-losing strategy, as anyone who has ever dealt with an IVR system knows. - Trustability is unrelated to size. There are one-person operations out there that are obviously far more trustable than huge ones. - Don't built what you can't control. Abuse handling needs to be factored into service offerings and growth decisions, not blown off and thereby forcibly delegated to the entire rest of the Internet. - Poorly-desigged and poorly-run operations markedly increase the workload for their own abuse desks. - A nominally competent abuse desk handles reports quickly and efficiently. A good abuse desk DOES NOT NEED all those reports because it already knows. (For example, large email providers should have large numbers of spamtraps scattered all over the 'net and should be using simple methods to correlate what arrives at them to provide themselves with an early heads up. This won't catch everything, of course, but it doesn't have to.) ---Rsk
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 8:34 AM, Rich Kulawiec [EMAIL PROTECTED] wrote: - Automation is far less important than clue. Attempting to compensate for lack of a sufficient number of sufficiently-intelligent, experienced, diligent staff with automation is a known-losing strategy, as anyone who has ever dealt with an IVR system knows. Rich, That is one place that modern antispam efforts fall apart. It's the same problem that afflicts tech support in general. The problem exists for the same reason that large-city McDonalds workers don't speak English: Anyone with sufficient clue to run an abuse desk is well qualified for more interesting, important and higher-paid work where they don't get yelled at all the time. Like administering mail servers or writing mail software. There's a reason we pay garbage collectors a small fortune to do a job that requires no skill whatsoever. Regards, Bill Herrin -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Apr 15, 2008, at 9:43 AM, William Herrin wrote: On Tue, Apr 15, 2008 at 8:34 AM, Rich Kulawiec [EMAIL PROTECTED] wrote: - Automation is far less important than clue. Attempting to compensate for lack of a sufficient number of sufficiently-intelligent, experienced, diligent staff with automation is a known-losing strategy, as anyone who has ever dealt with an IVR system knows. Rich, That is one place that modern antispam efforts fall apart. It's the same problem that afflicts tech support in general. The problem exists for the same reason that large-city McDonalds workers don't speak English: Anyone with sufficient clue to run an abuse desk is well qualified for more interesting, important and higher-paid work where they don't get yelled at all the time. Like administering mail servers or writing mail software. There's a reason we pay garbage collectors a small fortune to do a job that requires no skill whatsoever. Do you _know_ any garbage collectors ? I do, and I would disagree with both clauses of that sentence. Regards Marshall Regards, Bill Herrin -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 10:00 AM, Marshall Eubanks [EMAIL PROTECTED] wrote: On Apr 15, 2008, at 9:43 AM, William Herrin wrote: That is one place that modern antispam efforts fall apart. It's the same problem that afflicts tech support in general. The problem exists for the same reason that large-city McDonalds workers don't speak English: Anyone with sufficient clue to run an abuse desk is well qualified for more interesting, important and higher-paid work where they don't get yelled at all the time. Like administering mail servers or writing mail software. There's a reason we pay garbage collectors a small fortune to do a job that requires no skill whatsoever. Do you _know_ any garbage collectors ? I do, and I would disagree with both clauses of that sentence. Marshall, No, but I know a few people who have (briefly) worked abuse desks and neither the tech support nor the McDonalds problem are difficult to observe. Without conceding the garbage collection issue, let me ask you directly: how do you propose to motivate qualified folks to keep working the abuse desk? Regards, Bill Herrin -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Apr 15, 2008, at 10:31 AM, William Herrin wrote: On Tue, Apr 15, 2008 at 10:00 AM, Marshall Eubanks [EMAIL PROTECTED] wrote: On Apr 15, 2008, at 9:43 AM, William Herrin wrote: That is one place that modern antispam efforts fall apart. It's the same problem that afflicts tech support in general. The problem exists for the same reason that large-city McDonalds workers don't speak English: Anyone with sufficient clue to run an abuse desk is well qualified for more interesting, important and higher-paid work where they don't get yelled at all the time. Like administering mail servers or writing mail software. There's a reason we pay garbage collectors a small fortune to do a job that requires no skill whatsoever. Do you _know_ any garbage collectors ? I do, and I would disagree with both clauses of that sentence. Marshall, No, but I know a few people who have (briefly) worked abuse desks and neither the tech support nor the McDonalds problem are difficult to observe. Without conceding the garbage collection issue, let me ask you directly: how do you propose to motivate qualified folks to keep working the abuse desk? That is a good question. (I feel sure that many actually doing the job would opt for a rise in pay.) Maybe certain jobs should become apprentice-like positions that you need to get through to rise in a networking organization. I know that Craig Newmark (of Craig's List) spends a couple of hours per day going through abuse complaints and user issues personally. I haven't heard too many complaints about Craig's List, and it seems reasonable to suspect a connection there. That has the advantage of being cheap to implement, in dollars if not in political capital. Regards Marshall Regards, Bill Herrin -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 10:55 AM, Marshall Eubanks [EMAIL PROTECTED] wrote: On Apr 15, 2008, at 10:31 AM, William Herrin wrote: how do you propose to motivate qualified folks to keep working the abuse desk? That is a good question. (I feel sure that many actually doing the job would opt for a rise in pay.) Maybe certain jobs should become apprentice-like positions that you need to get through to rise in a networking organization. Marshall, There's a novel idea. Require incoming senior staff at an email company to work a month at the abuse desk before they can assume the duties for which they were hired. My hunch says that's a non-starter. It also doesn't keep qualified folks at the abuse desk; it shuffles them through. Any other ideas? Regards, Bill Herrin -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Abuse response [Was: RE: Yahoo Mail Update]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Joe Provo [EMAIL PROTECTED] wrote: It cannot be understated that even packet pushers and code grinders who care get stranded in companies where abuse handling is deemed by management to be a cost center that only saps resources. Paul, you are doing a serious disservice to those folks in specific, and working around such suit-induced damage in general, by dismissing any steps involving automation. Well, I did not intend to do disservice to anyone's efforts, but the point I am trying to make is that there still is no good way for people to report malicious activity to the legitimate owners of the content or the netblock. - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIBMyPq1pz9mNUZTMRAoiwAKDrdTSosQIT0r1BeRh2tvIQ5+at1QCgmS5W gdgRZ+CokBXlcfCehWtJKQg= =QDXi -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: Abuse response [Was: RE: Yahoo Mail Update]
William Herrin wrote: Without conceding the garbage collection issue, let me ask you directly: how do you propose to motivate qualified folks to keep working the abuse desk? Ask AOL? -Jack
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 11:22:59AM -0400, William Herrin wrote: There's a novel idea. Require incoming senior staff at an email company to work a month at the abuse desk before they can assume the duties for which they were hired. My hunch says that's a non-starter. It also doesn't keep qualified folks at the abuse desk; it shuffles them through. Require all technical staff and their management to work at the abuse desk on a rotating basis. This should provide them with ample motivation to develop effective methods for controlling abuse generation, thus reducing the requirement for abuse mitigation, thus reducing the time they have to spend doing it. ---Rsk
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Apr 15, 2008, at 10:33 AM, Rich Kulawiec wrote: On Tue, Apr 15, 2008 at 11:22:59AM -0400, William Herrin wrote: There's a novel idea. Require incoming senior staff at an email company to work a month at the abuse desk before they can assume the duties for which they were hired. My hunch says that's a non-starter. It also doesn't keep qualified folks at the abuse desk; it shuffles them through. Require all technical staff and their management to work at the abuse desk on a rotating basis. This should provide them with ample motivation to develop effective methods for controlling abuse generation, thus reducing the requirement for abuse mitigation, thus reducing the time they have to spend doing it. Unfortunately many of the skills required to be a competent abuse desk worker are quite specific to an abuse desk, and are not typically possessed by random technical staff. So, to bring this closer to nanog territory, it's a bit like saying that all the sales and customer support staff should be given enable access to your routers and encouraged to run them on a rotating basis, so that they understand the complexities of BGP and will better understand the impact their decisions will have on your peering. Cheers, Steve
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 10:56:02AM +0530, Suresh Ramasubramanian wrote: On Tue, Apr 15, 2008 at 10:16 AM, Paul Ferguson [EMAIL PROTECTED] wrote: As I mentioned in my presentation at NANOG 42 in San Jose, the biggest barrier we face in shrinking the time-to-exploit window with regards to contacting people responsible for assisting in mitigating malicious issues is finding someone to actually respond. Fergie.. you (and various others in the send emails, expect takedowns biz) - phish, IPR violations, whatever.. you're missing a huge, obvious point If you send manual notificattions (aka email to a crowded abuse queue) expect 24 - 72 hours response If you have high enough numbers of the stuff to report, do what large ISPs do among themselves, set up and offer an ARF'd / IODEF feedback loop or some other automated way to send complaints, that is machine parseable, and that's sent - by prior agreement - to a specific address where the ISP can process it, and quite probably prioritize it above all the j00 hxx0r3d m3 by doing dns lookups email. That kind of report can be handled within minutes. Is there an equivalent mechanism for those of us at the fringes of the galaxy to report problems? What is probably needed for little folks like me is not instant response but rather an address and formatting specs so that the information is of maximum usefullness to you and we don't get auto-naks. After all, I can probably generate a few reports a week, but not hundreds per day. -- -=[L]=- This work was funded by The Corporation for Public Bad Art despite their protestations.
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 2:04 PM, Steve Atkins [EMAIL PROTECTED] wrote: Unfortunately many of the skills required to be a competent abuse desk worker are quite specific to an abuse desk, and are not typically possessed by random technical staff. Steve, You don't, per chance, mean to suggest that random back-office technical staff might not have the temper and disposition to remain polite and helpful with the gentleman from the state capital so upset about the interdiction of his political mailings that he's ready to sic the regulators on you and wipe you off the map? The problem is that the individual who -does- have those skills along with the technical know-how to deal with the complaint itself usually ALSO has the skills to be the customer contact for a multi-million dollar contract. If you're a manager at a company that wants to, well, make money, which chair will you ask that individual to sit in? Regards, Bill -- William D. Herrin [EMAIL PROTECTED] [EMAIL PROTECTED] 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Apr 15, 2008, at 11:54 AM, William Herrin wrote: On Tue, Apr 15, 2008 at 2:04 PM, Steve Atkins [EMAIL PROTECTED] wrote: Unfortunately many of the skills required to be a competent abuse desk worker are quite specific to an abuse desk, and are not typically possessed by random technical staff. Steve, You don't, per chance, mean to suggest that random back-office technical staff might not have the temper and disposition to remain polite and helpful with the gentleman from the state capital so upset about the interdiction of his political mailings that he's ready to sic the regulators on you and wipe you off the map? The problem is that the individual who -does- have those skills along with the technical know-how to deal with the complaint itself usually ALSO has the skills to be the customer contact for a multi-million dollar contract. If you're a manager at a company that wants to, well, make money, which chair will you ask that individual to sit in? Not really. IMO, with decent automation[1] and a reasonably close working relationship between the abuse desk, the NOC and an internal sysadmin/developer or two, there's not that much need for a high level of technical know-how in the abuse desk staff. Good people skills are certainly important, and it'd be good to have at least one abuse desk staffer with a modicum of technical knowledge to handle basic technical questions, and help channel more complex ones to to NOC or developers efficiently, but the level of technical know-how needed to be an extremely effective abuse desk staffer is pretty low. The specific technical details they do need to know they can pick up from their peers (both within the abuse desk, in other groups of their company and, perhaps most importantly, from their peer at other companies abuse desks). It's closer to a customer support position, in skillset needed, than anything deeply technical, though an innate ability to remain calm under pressure is far more important in abuse than support. If you're big enough that you need more than one person staffing your abuse desk you can mix-n-match skills across the team too, of course. Cheers, Steve [1] Yeah, I develop abuse desk automation software, so I'm both reasonably exposed to practices at a range of ISPs and fairly biased in favor of good automation. :)
RE: Abuse response [Was: RE: Yahoo Mail Update]
So, to bring this closer to nanog territory, it's a bit like saying that all the sales and customer support staff should be given enable access to your routers and encouraged to run them on a rotating basis, so that they understand the complexities of BGP and will better understand the impact their decisions will have on your peering. We encourage managers, designers, engineers, project managers, etc. to spend a day handling customer support calls so that they understand the impacts of their decisions/work on the customer, who ultimately pays our paychecks. We run even more people through workshops where they spend some time listening to recorded customer support calls, and then plan how to prevent such problems in future so that the customers don't feel the need to call us. Of course, none of these people are expected to go in and reconfigure BGP sessions on routers, because there are working on first-line support. One of the duties of first-line support is to sift through the incoming and identify which cases need to be escalated to second or third-line support. Unless you have very good automated systems in place to ensure that the abuse desk only gets real cases to deal with, then you should be able to rotate managers and other employees through the abuse department to do some of that first-line sifting. If the outcome of this is that you make a business case for changes to abuse-desk systems and processes, then you should involve the abuse desk staff in this development work to give them some variety. Once those staff have automated themselves out of a job, you can move them to some other tools development project, or incident response work. --Michael Dillon
Re: Abuse response [Was: RE: Yahoo Mail Update]
On 15 Apr 2008, at 11:22 , William Herrin wrote: There's a novel idea. Require incoming senior staff at an email company to work a month at the abuse desk before they can assume the duties for which they were hired. At a long-previous employer we once toyed with the idea of having everybody in the (fairly small) operations and architecture/ development groups spend at least a day on the helpdesk every month. The downside to such a plan from the customer's perspective is that I'm pretty sure most of us would have been really bad helpdesk people. There's a lot of skill in dealing with end-users that is rarely reflected in the org chart or pay scale. Joe
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, 15 Apr 2008 19:14:52 EDT, Joe Abley said: The downside to such a plan from the customer's perspective is that I'm pretty sure most of us would have been really bad helpdesk people. There's a lot of skill in dealing with end-users that is rarely reflected in the org chart or pay scale. Of course - you're asking people who are *hired* because they're good at talking to inanimate objects made of melted sand, and asking them to relate to animate objects (namely, customers). Sounds like a recipe for disaster. :) pgphykYhcItQN.pgp Description: PGP signature
Re: Abuse response [Was: RE: Yahoo Mail Update]
Abuse desk is a $0 revenue operation. Is it not obvious what the issue is? Some of the folks that are complaining about abuse response generate revenue addressing these issues. Give me some of that. I'll give you a priority line to the NOC. Disclaimer; No offense intended to security providers, I'm just stating a fact. Best, Marty On 4/15/08, Joe Abley [EMAIL PROTECTED] wrote: On 15 Apr 2008, at 11:22 , William Herrin wrote: There's a novel idea. Require incoming senior staff at an email company to work a month at the abuse desk before they can assume the duties for which they were hired. At a long-previous employer we once toyed with the idea of having everybody in the (fairly small) operations and architecture/ development groups spend at least a day on the helpdesk every month. The downside to such a plan from the customer's perspective is that I'm pretty sure most of us would have been really bad helpdesk people. There's a lot of skill in dealing with end-users that is rarely reflected in the org chart or pay scale. Joe
Re: Abuse response [Was: RE: Yahoo Mail Update]
Abuse desk is a $0 revenue operation. Is it not obvious what the issue is? They're too busy spamming and phishing to respond to abuse reports? brandon
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, 2008-04-15 at 10:56 +0530, Suresh Ramasubramanian wrote: If you have high enough numbers of the stuff to report, do what large ISPs do among themselves, set up and offer an ARF'd / IODEF feedback loop or some other automated way to send complaints, that is machine parseable, and that's sent - by prior agreement - to a specific address where the ISP can process it, and quite probably prioritize it above all the j00 hxx0r3d m3 by doing dns lookups email. So how do the little guys play in this sandbox? My log files and spam reports are just as legit as the super-secret-handshake club guys are, and I'd like to get some respect. After all, I may be the first one to report it. Please keep a few things in mind though: - It needs to be simple to use. Web forms are a non-starter. - The output from any parsers needs to be human readable. There are too many auto-whatsit formatters for us to sit down and code to every one. - I'd like to see an actual response beyond an autoreply saying that you can't tell me who the customer is or what actions were taken. - I like dealing with other small operations and edus because humans actually do read the reports, and things get done (Thanks!). I've given up sending abuse reports to large consumer ISPs and all freemail providers because I'm not a member of the club. Any response that I'm lucky enough to get generally says something like You did not include the email headers in your complaint so we are closing this incident when I reported and FTP brute force. --Chris
Re: Abuse response [Was: RE: Yahoo Mail Update]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Paul Ferguson [EMAIL PROTECTED] wrote: Mow, this has no bearing on the original subject (which I have now forgotten what it is -- oh yeah, something about Yahoo! mail), but it should be additional proof that the Bad Guys know how to manipulate the system, the system is broken, and the Bad Guys are now making much more money than we are. :-) Actually, that was supposed to read: Meow, this has no bearing... Just kidding. :-) http://imdb.com/title/tt0247745/ - - ferg p.s. I guess we should all lighten up a little and actually figure out out to do abuse notification/communications a bit better. Meow. -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIBDq/q1pz9mNUZTMRAos2AJ9Rv3jRNc3Dmx/31Vtk8p3y0MTJ+QCfc2z8 kM2w7GkCJVc2WU6dbsp0+FI= =cp/T -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: Abuse response [Was: RE: Yahoo Mail Update]
On Tue, Apr 15, 2008 at 10:16 AM, Paul Ferguson [EMAIL PROTECTED] wrote: As I mentioned in my presentation at NANOG 42 in San Jose, the biggest barrier we face in shrinking the time-to-exploit window with regards to contacting people responsible for assisting in mitigating malicious issues is finding someone to actually respond. Fergie.. you (and various others in the send emails, expect takedowns biz) - phish, IPR violations, whatever.. you're missing a huge, obvious point If you send manual notificattions (aka email to a crowded abuse queue) expect 24 - 72 hours response If you have high enough numbers of the stuff to report, do what large ISPs do among themselves, set up and offer an ARF'd / IODEF feedback loop or some other automated way to send complaints, that is machine parseable, and that's sent - by prior agreement - to a specific address where the ISP can process it, and quite probably prioritize it above all the j00 hxx0r3d m3 by doing dns lookups email. That kind of report can be handled within minutes. If you send reports with lots of legal boilerplate, or reports with long lectures on why you expect an INSTANT TAKEDOWN, and send them to a busy abuse queue, there is no way - and zero reason - for the ISP people to prioritize your complaint above all the other complaints coming in. Unfortunately, most abuse requests/inquiries fall into a black-hole, or bounce. Not you, but several companies that do this as a business model need to learn how to do this properly. Some of them are spectacularly incompetent at what they do too. Me, I have pretty much given up on any domain-related avenues, since they generally end up in disappointment, and found more successes in going directly to the owners of the IP allocation, and upstream ISP, a regional/national CERT/CSIRT, or law enforcement. Yeah? And by the time your request filters right back down to where it actualy belongs.. guess what, it takes much longer than 72 hours. Mow, this has no bearing on the original subject (which I have now forgotten what it is -- oh yeah, something about Yahoo! mail), but it should be additional proof that the Bad Guys know how to manipulate the system, the system is broken, and the Bad Guys are now making much more money than we are. :-) And proof that various good guys dont know how to cooperate, and various other good guys are in the business only to score points off other providers to make themselves look good. http://blog.washingtonpost.com/securityfix/2007/12/top_10_best_worst_antiphishing.html for example.. I think Brian Krebs - given what I know of his usual high standards - would certainly have regretted publishing PR and marketing generated, highly debatable, statistics like the ones referenced in that article. --srs
Re: Abuse response [Was: RE: Yahoo Mail Update]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Suresh Ramasubramanian [EMAIL PROTECTED] wrote: If you send reports with lots of legal boilerplate, or reports with long lectures on why you expect an INSTANT TAKEDOWN, and send them to a busy abuse queue, there is no way - and zero reason - for the ISP people to prioritize your complaint above all the other complaints coming in. In fact, we have done just that -- develop a standard boilerplate very similar to what PIRT uses in its notification(s) to the stakeholders in phishing incidents. Again, our success rate is somewhere in the 50% neighborhood. And that is after a few months of fine-tuning -- and 15 years of experience in these matters. :-) Nothing to write home about... - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFIBD5wq1pz9mNUZTMRAtyzAJ9yeVdLNPQYgCoacK5sNwe3N9xZ9QCeLSlS /JALeFX6VwD6Qb430CSt6yI= =f3fI -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
