Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On maandag, sep 8, 2003, at 17:30 Europe/Amsterdam, Dean Anderson wrote: Nobody cares. Making a roof 100.00% impervious to water molecules may be impossible, but that doesn't mean we have to resign to getting wet every time it rains. People care because when someone comes around saying you can have a 100% impervious roof if only you jump through these inconvenient hoops, we know that they are wrong, and don't need to waste time considering how inconvenient the hoops are. Your analogy doesn't fly. Our email protocols have holes big enough to drive a truck through. Is it unreasonable when people ask the IETF leadership for a place to work on this? We, meaning the IETF, care, because this is very useful aid to deciding what to work on. We know that we need to focus on leak stoppage, not trying to invent leak-proof protocols. There is no point researching something that is impossible. Let's first define our goal before declaring it impossible to reach. After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. Where does this covert channel stuff come from anyway? What do you mean? The jump from spam to covert channel isn't immediately obvious. And if any proof of why spam is a covert channel has been offered, I've managed to miss it. the spammer's achilles heel is that they need to send out incredible amounts of email in order to fulfill their objectives, whichever those are. Detecting bulk mail is doable, and it shouldn't be too hard to come up with something to differentiate legitimate bulk emailing from spam. For instance, we can reverse the burden of proof here and only allow know bulk emailers. Detecting abuse is quite different from making a protocol that can't be abused. If you can detect abuse on input rather than on output, detecting abuse is exactly what enables you to make an abuse-free protocol. And again: just because there are situations that you can't do something doesn't mean it's automatically useless to perform the action under other circumstances too. But that is my point: You have to focus on detection. This doesn't require any protocol changes whatsover. Do you mean first accepting the message, then searching for anatomical, pharmaceutical or financial terms and subsequently discarding the message? This is useless as it only makes the spammers spam more, not less. We are already only allowing known bulk emailers. Untrue. Unfortunately, that doesn't prevent spam. It should help if used together with some kind of authentication (for which, I'm happy to see, there are several independently submitted proposals on the table) so that spammers can't inject arbitrary from addresses. Indeed, it seems most of the spam isn't commercial: Most of the spam seems to come from viruses, and isn't really selling anything. The point being? The viruses can use the credentials of the infected user. That is legitimate, until someone reading the email realizes its not and complains. These send 40-50 messages per IP, and is hard to detect as bulk. But when added up over a lot of IP addresses, is quite obviously annoying. Not as annoying as real spam. (Although the size of email worms is unpleasant. Fortunately, regular spam is usually pretty small.) Fixable with authentication. No, that's the point. It isn't _fixable_ with authentication. It isn't fixable at all. It is only fixed when the spammer loses his account. Then the spammer gets a new account. So it isn't really fixed. It is when they lose their accounts faster than they can acquire new ones. So we are always going to be playing a game of whack-a-mole. So? I don't mind doing that if I can win. Having to play and then be forced to lose is much harder to swallow. That cannot be avoided by altering the protocol or the authentication scheme (information theory proves this). So it is useful, then, to work on ways of detection, and improve our whack-a-mole skills. Altering protocols and authentication is a waste of time. I find it unacceptable that my email software lets people bombard me with crap while at the same time faking the headers such that said crap seems to come from an innocent bystander. I can't understand how any reasonable person can think this is ok. Or maybe we should just go ahead and move the From: header to historic and be done with it.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
The viruses can use the credentials of the infected user. That is legitimate, until someone reading the email realizes its not and complains. These send 40-50 messages per IP, and is hard to detect as bulk. Reports from some operators of DCC clients at non-trivial sites claim that the DCC does a tolerable job against SoBig.F. This is without the Greylist support now available in the DCC client code. The DCC detects bulk mail, defined as substantially identical messages from any SMTP client senders. I'd not expect the DCC to do well against most worms or viruses. SoBig is somewhat different. (I won't talk those differences in public or with people I don't know well enough to say they'll also be descrete. Like other people who care more about fighting viruses and spam than being known as fighters of viruses and spam, I think the profit in idle chatter is not worth the cost of giving even trivial aid and comfort to the bad guys.) As has been pointed out, all of this belongs in the ASRG mailing list if anywhere. See http://irtf.org/charters/asrg.html and https://www1.ietf.org/mail-archive/working-groups/asrg/current/maillist.html Vernon Schryver[EMAIL PROTECTED]
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Why is this even difficult. I have yet to see a firm proposal (ie. an Internet Draft), and once there is one, it is a simple matter of asking an AD to sponsor a BOF to see if there is interest in forming a working group to solve the problem. I remember sitting through several YATP (Yet another Tunnelling Protocol) BOFs years ago, I don't see what the problem is with creating some YASPP BOFs (Yet Another Spam Prevention Protocol). This is the IETF, that is the IETF process... Why are we arguing here about killing it without having a firm proposal, and a BOF chartered to form a WG to go solve the problem. Any more breath we waste now doesn't help anybody. My challenge - Go forth - publish your protocol in ID form, contact the correct APP (probably) area AD and get a BOF in Minneapolis - Convince the IETF in general there that a WG should be chartered to solve the problem. Go and solve the problem Bill On Tue, Sep 09, 2003 at 01:41:33PM -0400, Dean Anderson wrote: My apologies for this message. This discussion is winding down. Iljitsch makes some interesting points, to which I have tried to respond thoughtfully. --Dean On Tue, 9 Sep 2003, Iljitsch van Beijnum wrote: On maandag, sep 8, 2003, at 17:30 Europe/Amsterdam, Dean Anderson wrote: Nobody cares. Making a roof 100.00% impervious to water molecules may be impossible, but that doesn't mean we have to resign to getting wet every time it rains. People care because when someone comes around saying you can have a 100% impervious roof if only you jump through these inconvenient hoops, we know that they are wrong, and don't need to waste time considering how inconvenient the hoops are. Your analogy doesn't fly. Our email protocols have holes big enough to drive a truck through. Is it unreasonable when people ask the IETF leadership for a place to work on this? I don't think our email protocols have any holes at all. They can be abused. But mere abuse is not a hole. We, meaning the IETF, care, because this is very useful aid to deciding what to work on. We know that we need to focus on leak stoppage, not trying to invent leak-proof protocols. There is no point researching something that is impossible. Let's first define our goal before declaring it impossible to reach. Well, I think the goal has been stated: Create an abuse-free email protocol. That goal is impossible. Thus, we have abusable protocols. Perhaps you have a different goal in mind, but it doesn't sound like you accept the premise that it impossible to create an abuse-free protocol. After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. Where does this covert channel stuff come from anyway? What do you mean? The jump from spam to covert channel isn't immediately obvious. And if any proof of why spam is a covert channel has been offered, I've managed to miss it. The NCSC's definition refers to ANY communication not authorized by the security model. Note that the term Covert Channel is frequently associated with Multilevel Secure Operating Systems. The liturature uses other terms to describe the same concept: information leakage, sneaky signalling, hidden data flows, side channels. So you must not get too caught up in the peculiarities of operating systems. The concept is quite general. It is immediately obvious that covert channels have 3 characteristics: It is a CHANNEL, it is COVERT, and it VIOLATES SECURITY POLICY. (Caps for emphasis of definition, not loudness.) CHANNEL: Spam is a type of Email. Email is a channel transfering signals from A to B. Email is really a subchannel of the internet, which is a subchannel of the PSTN, public and private fiber networks, etc. COVERT: Spam is hidden in so far as possible from the system operators. It is an unintended communication in that the system operators intended that only non-broadcast, solicited commercial communication flow. This not an exclusive list of what is permitted, but illustrates that spam isn't permitted. VIOLATES SECURITY POLICY: System Operators specified an acceptable use policy that outlines what is permitted and what is not permitted. UCE is not permitted. Various methods have been implemented to enforce this policy. the spammer's achilles heel is that they need to send out incredible amounts of email in order to fulfill their objectives, whichever those are. Detecting bulk mail is doable, and it shouldn't be too hard to come up with something to differentiate legitimate bulk emailing from spam. For instance, we can reverse the burden of proof here and only allow know bulk emailers. Detecting abuse is quite different from making a protocol that can't be abused. If you can detect abuse on input rather than on output, detecting
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 01:41 PM 9/9/2003 -0400, you wrote: My apologies for this message. This discussion is winding down. Iljitsch makes some interesting points, to which I have tried to respond thoughtfully. Dean, Yes as already stated, I do intend to close this thread and eventually provide a forwarding link to a new discussion elsewhere (perhaps at IRTF as someone suggested, if possible...) However, I think the analysis of the concepts of information theory, channels, and models of spam is more fundamental to internet engineering than the original purpose of this thread and thus I see no reason why it would not be useful data here at IETF. Before I respond to your continuance of your argument, I *respectfully* remind that I already refuted the whole line of criticism you are continuing in this post, when I rebutted your last post in this thread: http://www1.ietf.org/mail-archive/ietf/Current/msg22139.html In case any one missed it, I think the most relevant section there begins done the page a bit with: Disagree strongly. First benefit is once you define spam == *BE (instead of UBE), then it is easier to model spam... more below... Your analogy doesn't fly. Our email protocols have holes big enough to drive a truck through. Is it unreasonable when people ask the IETF leadership for a place to work on this? I don't think our email protocols have any holes at all. They can be abused. But mere abuse is not a hole. Semantics debate only. Better to stick to the real point below... We, meaning the IETF, care, because this is very useful aid to deciding what to work on. We know that we need to focus on leak stoppage, not trying to invent leak-proof protocols. There is no point researching something that is impossible. Let's first define our goal before declaring it impossible to reach. Well, I think the goal has been stated: Create an abuse-free email protocol. No that is not the stated goal of this thread I started. I already rebutted that whole link of criticism here: http://www1.ietf.org/mail-archive/ietf/Current/msg22139.html Look for the section that starts with: Your point is that it is futile to define a protocol... And here: http://www1.ietf.org/mail-archive/ietf/Current/msg22129.html Start reading down from: I proposed an way to improve leak stoppage, by defining the signal in the channel and not only at end points. I never proposed a leak-proof protocol. Perhaps you have a different goal in mind, but it doesn't sound like you accept the premise that it impossible to create an abuse-free protocol. The links to the previous posts are above which state that is not our goal. You have been told that at least 2 or 3 times already. Iljitsch van Beijnum wrote: The jump from spam to covert channel isn't immediately obvious. And if any proof of why spam is a covert channel has been offered, I've managed to miss it. Iljitsch van Beijnum, I think what Dean Anderson means is that because you can't create a 100% perfect covert channel, then spammers will always find a way to abuse, no matter what you do on the protocol level. Theoretically I agree with him. However, he is ignoring the posts I made (as linked above), which show that is not what I am proposing. What I am proposing has to do with improving the model of spam so it can be more easily detected at more points in the channels and earlier and other detection advantages. To get this model, I propose that we need a new definition of legitimate bulk email, from push to pullrather than repeat my entire logic here, please read the linked posts above in entirety. Dean Anderson wrote: The NCSC's definition refers to ANY communication not authorized by the security model. Note that the term Covert Channel is frequently associated with Multilevel Secure Operating Systems. The liturature uses other terms to describe the same concept: information leakage, sneaky signalling, hidden data flows, side channels. So you must not get too caught up in the peculiarities of operating systems. The concept is quite general. And COVERT has nothing to do with my proposal as I've detailed ad nauseum in the above linked posts. CHANNEL: Spam is a type of Email. Email is a channel transfering signals from A to B. Email is really a subchannel of the internet, which is a subchannel of the PSTN, public and private fiber networks, etc. And moving legitimate bulk email to a pull channel is part of my proposal. COVERT: Spam is hidden in so far as possible from the system operators. It is an unintended communication in that the system operators intended that only non-broadcast, solicited commercial communication flow. This not an exclusive list of what is permitted, but illustrates that spam isn't permitted. Part of my overall point made in the links of posts above is that one of the reasons it is hidden is because it can only currently be modeled pyschologically, because the definition is UBE (unsolicited bulk
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I started this thread. Let's please close this thread here on IETF and move it some where more appropriate. I do not know exactly where to move it to (which list I will choose exactly because I want to research more to try to avoid running into Vernon, Valdis, Spencer, and others again which will be difficult since they seem to be every where). What I may do is write the LD and make a working group some where and then apply to such a list and ask only those are really interested to follow. Sometime in the future, I will make one more post on this list to say where all my replies will be going. So if there are any more posts, I will be collecting them and making replies on them to another list in future. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Very nice. I say to post an Internet Draft - you post a link to a simple archived e-mail. The IETF process starts with an Internet Draft - without it we are all just wasting time. An internet draft is a concrete proposal that can be discussed, archived, debated successfully, etc. I challenge you to take your e-mail posting and turn it into an Internet Draft with a legitimate security section (since you are solving a security problem) then post a single message to this list showing the internet draft, and a mailing list that people can join if they are interested in discussing it further. From there it is easy to determine if there is enough interest in forming a BOF in Minnesota ( or S. Korea in March ) to forward the work in a Working Group. The problem you have run into is you haven't taken the first step, which is to simply submit an Internet Draft to the Internet Drafts editor... very simple process with no politics in the way. From there you can pursue forming a Working Group (where you will get your first taste of politics). Being that you haven't taken the first step (publishing an Internet Draft) I am not sure you really meet the charter (Ok, yes you do, but putting out a draft is SO important - it should be the first step) and you have allowed the topic to grow WAY beyond initial discussions (I am actually waiting for Harold to post one of his famous Top n Talker lists soon). The next step is to get a mailing list somewhere and move the discussion off of this list onto that list Bill On Wed, Sep 10, 2003 at 05:10:06AM +0800, Shelby Moore wrote: Why is this even difficult. I have yet to see a firm proposal (ie. an Internet Draft),... My challenge - Go forth - publish your protocol in ID form... 1. I remind you to read my initial post that started this thread: http://www1.ietf.org/mail-archive/ietf/Current/msg22035.html Request for opinions on whether to creating a working group or publish the following idea as an internet draft? I think that qualifes under initial discussion of the charter of this list (see #2 below). 2. And to read the charter for this mailing list: http://www.ietf.org/maillist.html This list is meant for initial discussion only. Discussions that fall within the area of any working group or well established list should be moved to such more specific forum... 3. Just a fews posts back in this thread, it was suggested that IRTF would be a better forum for anti-spam proposals and discussions, and I agreed (to consider it if possible and applicable). However there is a another line of discussion in this thread pertaining to general information theory and modeling of channels which is still winding down (initial discussion) and seems quite general to internet engineering. 4. The basic difficulty has been those violating the charter of the list: http://www.ietf.org/maillist.html Unprofessional commentary, regardless of the general subject. such as the Kook thread offshoot that someone started. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On dinsdag, sep 9, 2003, at 19:41 Europe/Amsterdam, Dean Anderson wrote: Let's first define our goal before declaring it impossible to reach. Well, I think the goal has been stated: Create an abuse-free email protocol. That goal is impossible. Thus, we have abusable protocols. Ok, not going to argue on that point (not necessarily agreeing, though). Perhaps you have a different goal in mind, but it doesn't sound like you accept the premise that it impossible to create an abuse-free protocol. My goal is to drastically lower unsullicited bulk mail. It's not the occasional message that bothers me, but the dozens each day that try to exploit human kind's lowest common denominator. I believe bulk and unsollicited can both be determined objectively, so there should be ample avenues to explore here. The NCSC's definition refers to ANY communication not authorized by the security model. Note that the term Covert Channel is frequently associated with Multilevel Secure Operating Systems. The liturature uses other terms to describe the same concept: information leakage, sneaky signalling, hidden data flows, side channels. So you must not get too caught up in the peculiarities of operating systems. The concept is quite general. It is immediately obvious that covert channels have 3 characteristics: It is a CHANNEL, it is COVERT, and it VIOLATES SECURITY POLICY. (Caps for emphasis of definition, not loudness.) Ok. CHANNEL: Spam is a type of Email. Email is a channel transfering signals from A to B. Ok. COVERT: Spam is hidden in so far as possible from the system operators. It is an unintended communication in that the system operators intended that only non-broadcast, solicited commercial communication flow. This not an exclusive list of what is permitted, but illustrates that spam isn't permitted. Hm, is it? In most cases spamminess is pretty obvious, but I've both received and sent out complaints about spam which turned out to be fairly legitimate communication that just wasn't all that welcome any more. (But hard to tell when businesses change their (domain) name.) This stuff is hard to determine on a per-message basis. The part that I care about is that spam is unwelcome in two ways: each individual spam message is unwanted by the recipient, and the generated system load is unwanted by the service provider. VIOLATES SECURITY POLICY: System Operators specified an acceptable use policy that outlines what is permitted and what is not permitted. UCE is not permitted. Various methods have been implemented to enforce this policy. Again, too hard to determine by just looking at a message somewhere in the middle of the path. Now if we define that a single user may only send 1000 messages a day, that's something we can work with. If you can detect abuse on input rather than on output, detecting abuse is exactly what enables you to make an abuse-free protocol. Input to the first mailserver is output to the mail client. There may be many input/output connections. For every input, there is an equal and opposite output. (Seems that perpetual motion may have some analog after all ;-) Yes, this makes the whole thing somewhat harder... We are already only allowing known bulk emailers. Untrue. Speak for yourself. You are the one who said we. :-) I do think Shelby was on the right track with the idea of separating mailinglists from regular one-to-few email. (Not quite sold on the pop thing, though, that would be going 8 years back in time for me.) Mailinglists usually already have reasonable access controls and a single or very limited number of sources, so those shouldn't be a huge problem. With the sollicited bulk email out of the way, any bulk email that's left is automatically unsollicited (or a corner case that we can ignore at the price of some inconvenience) so this can be stopped or slowed down. Spammers can _always_ do whatever regular users can do. There is no way to authenticate regular users and not authenticate spammers. This is why SMTP AUTH has no effect on spam. If I know who sent something I can filter so I don't have to receive any future filth from the same source. And if spammers are forced to use their real identity then finally they'll get at least some of what they have coming. If regular users can inject arbitrary addresses, then so can spammers. Regular users can always inject arbitrary addresses. Ordinarilly, this is desired. Disagree. The point being? The point being that the persons sending most of the spam will do anything to annoy people. It has no grounding in commercial motivation, nor can regulation of commercial activity have any effect. In other words, protocols that require cooperation of the spammer aren't likely to succeed. What we need is a system where the only action that has any result is a valid action. Then either spammers have to spew out their filth out in the open without being able to hide, and bear the consequences
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 7 Sep 2003, Iljitsch van Beijnum wrote: On zondag, sep 7, 2003, at 21:45 Europe/Amsterdam, Dean Anderson wrote: Information theory says that such things are impossible. One can not construct a spam-free protocol because this is the same problem as constructing a system free of covert channels, which information theory says is impossible. Nobody cares. Making a roof 100.00% impervious to water molecules may be impossible, but that doesn't mean we have to resign to getting wet every time it rains. People care because when someone comes around saying you can have a 100% impervious roof if only you jump through these inconvenient hoops, we know that they are wrong, and don't need to waste time considering how inconvenient the hoops are. We, meaning the IETF, care, because this is very useful aid to deciding what to work on. We know that we need to focus on leak stoppage, not trying to invent leak-proof protocols. There is no point researching something that is impossible. It is not simply hard. It is impossible, like perpetual motion. So when exactly was the earth supposed to stop moving? God didn't make the earth move perpetually. He just made it move long enough. It seems that even God can't solve some problems. We didn't get to the moon by inventing perpetual motion machines, though early proposals were based on such machines. We got to the moon by working on the messy physics of rockets. When someone comes to the NSF and says you can have a perpetual motion machine if only you jump through some very inconvenient hoops, and spend a lot of money, the NSF can save itself the time and money by discarding perpetual motion schemes from its research program. Similarly, information theory allows us to discard some ideas from our research programs. That is why we care. After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. Where does this covert channel stuff come from anyway? What do you mean? But this is a simpler way to think about it: Spammers can continue to claim they are legitimate emailers, because they _ARE_ legitimate, so far as we know before they send email. And even so far as we know _before_ someone _READS_ their email. Only after reading their email, and perhaps only after some investigation, can we know for sure that the sender and message is conducting abuse or in violation of their AUP. This goes for each individual message, but the spammer's achilles heel is that they need to send out incredible amounts of email in order to fulfill their objectives, whichever those are. Detecting bulk mail is doable, and it shouldn't be too hard to come up with something to differentiate legitimate bulk emailing from spam. For instance, we can reverse the burden of proof here and only allow know bulk emailers. Detecting abuse is quite different from making a protocol that can't be abused. But that is my point: You have to focus on detection. This doesn't require any protocol changes whatsover. We are already only allowing known bulk emailers. Unfortunately, that doesn't prevent spam. Indeed, it seems most of the spam isn't commercial: Most of the spam seems to come from viruses, and isn't really selling anything. The viruses can use the credentials of the infected user. That is legitimate, until someone reading the email realizes its not and complains. These send 40-50 messages per IP, and is hard to detect as bulk. But when added up over a lot of IP addresses, is quite obviously annoying. It is not immune to spam, though it distributes spam and other broadcast messages much more efficiently than typical email systems. Ouch! :-) Fixable with authentication. No, that's the point. It isn't _fixable_ with authentication. It isn't fixable at all. It is only fixed when the spammer loses his account. Then the spammer gets a new account. So it isn't really fixed. So we are always going to be playing a game of whack-a-mole. That cannot be avoided by altering the protocol or the authentication scheme (information theory proves this). So it is useful, then, to work on ways of detection, and improve our whack-a-mole skills. Altering protocols and authentication is a waste of time. --Dean
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Information theory says that such things are impossible. One can not construct a spam-free protocol because this is the same problem as constructing a system free of covert channels, which information theory says is impossible. But information theory also says you can optimize signal-to-noise ratio, but only if you know what the characteristics of your signal are. Thus my whole motivation for an unambiguous definition (spam == all bulk email) along the channel and not just a definition at the end points (UBE). Nobody cares. Making a roof 100.00% impervious to water molecules may be impossible, but that doesn't mean we have to resign to getting wet every time it rains. People care because when someone comes around saying you can have a 100% impervious roof if only you jump through these inconvenient hoops, Who said 100%? Is the problem already reduced to an acceptable S/N ratio for the majority? Is the current S/N ratio a problem for the majority? we know that they are wrong, How can you know they are wrong, even you did not even realize that no one was proposing a 100% solution in this thread?? We, meaning the IETF, care, because this is very useful aid to deciding what to work on. You decline work without even understanding what is being written about a new idea? No one ever proposed 100% solution. I challenge you to find one post where I wrote that. We know that we need to focus on leak stoppage, not trying to invent leak-proof protocols. I proposed an way to improve leak stoppage, by defining the signal in the channel and not only at end points. I never proposed a leak-proof protocol. Underlying in what you are saying is you don't support new protocols, because you have a vested interest in existing methods of detection. We didn't get to the moon by inventing perpetual motion machines, And many people said it was impossible for us to get to the moon. though early proposals were based on such machines. We got to the moon by working on the messy physics of rockets. What about the messy information theory of defining the spam signal for the channel and not just the end points, so that it can be studied and research in the channel and not only in the receiver's mind Have you not realized yet that profound point I am making??? After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. Where does this covert channel stuff come from anyway? What do you mean? The use of covert here probably meant that making anything 100% covert is impossible in information theory. But this is a simpler way to think about it: Spammers can continue to claim they are legitimate emailers, because they _ARE_ legitimate, so far as we know before they send email. And even so far as we know _before_ someone _READS_ their email. Only after reading their email, and perhaps only after some investigation, can we know for sure that the sender and message is conducting abuse or in violation of their AUP. You exactly stated the problem. We have not defined spam (the signal we need to sample) in terms of the channel. If you define it as ALL BULK EMAIL, then you can actually start some messy science of getting to the moon on spam issue... This goes for each individual message, but the spammer's achilles heel is that they need to send out incredible amounts of email in order to fulfill their objectives, whichever those are. Detecting bulk mail is doable, and it shouldn't be too hard to come up with something to differentiate legitimate bulk emailing from spam. For instance, we can reverse the burden of proof here and only allow know bulk emailers. Detecting abuse is quite different from making a protocol that can't be abused. But that is my point: You have to focus on detection. This doesn't require any protocol changes whatsover. You can not measure a signal if you have not defined it. That is a fundamental concept of information theory. Spam is ambigous in the channel, unless you define spam == all bulk email. How many dozens of times have I written that in this thread! We are already only allowing known bulk emailers. Unfortunately, that doesn't prevent spam. SOBO (statement of blatantly obvious) that you can't filter something if you can't define it. That information theory 101. Indeed, it seems most of the spam isn't commercial: Most of the spam seems to come from viruses, and isn't really selling anything. The viruses can use the credentials of the infected user. That is legitimate, until someone reading the email realizes its not and complains. These send 40-50 messages per IP, and is hard to detect as bulk. Viruses are a different signal and can be filtered as such, unless they contain no viral data. No, that's the point. It isn't _fixable_ with authentication. It isn't fixable at all. It is
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I am not talking about email spreading virues. A number of viruses appear to send spam. (not spreading). Sometimes this is autonymous. Sometime it is under control via IRC channel back to the virus operator. Further, it seems that many open proxies are installed by virus. Once the virus has control of the computer, it has or will obtain keys to private keychains, etc. It can do whatever the infected users can do. The number of 40-50 emails per IP figure comes from analysis of spam messages that get by filters, by reviewing how many messages came from the same source. A lot of spam that gets by filters is of this very low volume type. --Dean On Tue, 9 Sep 2003, Shelby Moore wrote: Indeed, it seems most of the spam isn't commercial: Most of the spam seems to come from viruses, and isn't really selling anything. The viruses can use the credentials of the infected user. That is legitimate, until someone reading the email realizes its not and complains. These send 40-50 messages per IP, and is hard to detect as bulk. This is pseudo-off topic because I already stated below that a viral signal can be detected differently than a spam signal, unless it contains no viral data (which would be pointless afaik). I am curious about your data. Are you refering to emails spreading a virus that contain viral attachments?? It occurs to me that a virus can not spread very fast or effectively if each infected computer only sends 50 emails, because the infection rate is probably similar to spam, i.e. 0.005%. So you would only get 1 new infection for each 20,000 emails sent, or thus for each 400 infected computers. It seems the virus would likely die (anti-virus actions) at that rate of spread. So I must assume you were looking at a very small sample on internet email and you did not extrapolate??? Your answers might be somewhat helpful to me in my work. Thanks, Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I am not talking about email spreading virues. A number of viruses appear to send spam. (not spreading). Sometimes this is autonymous. Sometime it is under control via IRC channel back to the virus operator. Further, it seems that many open proxies are installed by virus. Once the virus has control of the computer, it has or will obtain keys to private keychains, etc. It can do whatever the infected users can do. The number of 40-50 emails per IP figure comes from analysis of spam messages that get by filters, by reviewing how many messages came from the same source. A lot of spam that gets by filters is of this very low volume type. Dean this is important data for me, but not as useful until I can get you to tell me: 1. What was the time duration of the sample? 2. What was your total approximate sample size? 3. What was the total volume of this type of spam within that sample? Estimates would be fine. I would really appreciate it very much. Thanks, Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
After this issue, I am probably moving the thread to IRTF (as suggested) if possible (but probably after taking a break to do some other work). Information theory says that such things are impossible. One can not construct a spam-free protocol because this is the same problem as constructing a system free of covert channels, which information theory says is impossible. But information theory also says you can optimize signal-to-noise ratio, but only if you know what the characteristics of your signal are. It actually doesn't say that precisely. It says that you can transmit a signal with an arbitrarilly low error rate at a speed below the channel capacity. The concrete task of altering the signal to noise ratio is accomplished by enhancing the signal with a harmonic oscillator, so that it is stronger than the noise. Agreed. And thus on a conceptual level, you have to have some idea about the signal characteristics in order to enhance it. Actually if I remember correctly, your example is how it applies to periodic signals. The general case is more abstract. This is then described as a set of differential equations that can be optimized with Variational methods. The limits of this process are indicated by information theory, the nyquist theorem, etc. Add Shannon entropy, chaos, etc... If the channel isn't described by a fourier series, then the differential equations may not be solvable, and it may be impossible to optimize its signal to noise ratio. (Well, there are other mathematical methods, but you get the point.) Yes that is what I meant that the general case is more abstract, so I was talking on a conceptual or abstract level. You are borrowing the concepts by metaphor, but the concrete methods don't transfer well. I was only using it to say we must define the signal how it appears in the channel before we can do any research on it in the channel. The way spam is currently defined defined as UBE (instead of my proposed *BE), then it means you can only model the signal at the end point. Given that means in the receivers subjective mind, that is not all that useful for research, unless you want to get into very fuzzy science such pyschology. If you want to make the point about practicality, then that is a very strong one! My point is not to discourage you from trying to stop spam, You are only 1 of 3 people so far at IETF who has said that to me. The rest who have commented have tried to discourage me. So thank you. but to focus your attention on detection, rather than protocol alteration. It is impossible to alter the protocol in any way that will force the spammer to identify themselves a-priori as a spammer. Disagree strongly. First benefit is once you define spam == *BE (instead of UBE), then it is easier to model spam and do research on it, because you can model it at any node in the channel, not only at the receiver end point. That was my whole point about enforcers. However, there is a problem. Some *BE is solicited. Which is why I proposed moving the solicited *BE to another channel (pull). Your point is that it is futile to define a protocol that will separate the solicited from the unsolicited, because spammers will always be able to subvert the protocol. And you to say thus there are no benefits to detection. I strongly disagree. There are two aspects to my response: 1. Spam coming thru the alternate pull channel can be modeled differently that spam defined as *BE. This separation of models provides benefits over trying to model spam as UBE in the receiver's mind (end point). Other person in this thread has provided one specific example, which is the pull delay gives a whole new dynamic to detection. Also I have pointed about that the membership quality of the solicited channel, gives it unique modeling advantages. 2. Spam coming thru the existing channel can then be modeled as *BE at any node of the channel, instead of as UBE. Some nodes have a much better model of spam in this definition, than the one at the end point. For example, ISPs can see a lot more abuse data in real-time, than a single receiver or the current inherently more clumsy attempts to group or poll receivers. Hopefully that will set the record straight that I am thinking about spam in new conceptual ways...and not rehashing as others have claimed... You could ask for spammers to cooperatively self-mark their messages. But this hasn't been terribly productive. Obviously I am not asking for that or any thing like that. See above. It is also pointless to ask for cooperative identification of non-spammers and identify spammers as those not in the set of non-spammers. I am also not asking for this, and it is instructive to understand how I am not. I am only making a definition, so that one can model under the benefits of that definition. What people actually do is a different matter, but as I pointed out
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
IMO, this (whether Hotmail will implement a specific feature) is a fairly irrelevant (an 80 out of 80/20 rule) fork of the debate relative to the main point of the proposal, so let's try to wrap this fork up with one or two go rounds max okay. Interestingly note that Hotmail makes you pay to POP *FROM* hotmail, but no charge to POP from other accounts *TO* Hotmail. Does that give you any hint about their business model?? Yes. It's *NOT* a business model where they want to be polling a dozen servers on a regular basis for each of their customers for mail that may or may not be there, and for the average mailing list, probably is not there at any given poll. Not any more than they want to be POPing any email at all. Nobody wants to do any work they do not have to do. But if there is advantage for them, or a profit to be made, they do it. If they do not, someone else will, then they lose marketshare. They used to not POP email at all (do you remember or did you know that?). Then they discovered they were missing a big market of eyeballs. They want eyeballs, and the last thing they want to do is expend more effort than needed to get eyeballs. No disrespect intended, but that sentence is illogical. You want something and you don't want to do something that gives you what you want. You mean I guess that they would not agree to add effort just to retain existing eyeballs. Again I disagree. I think they will do what ever they have to in order to retain market share, as long as the cost doesn't kill more profit than it retains. Sure - they can even optimize the 'POP the list' check by only doing it once for all the subscribers - but they're still hitting each server for each list on a several-times a day basis. And under the current scheme, they can just *catch* one SMTP transaction with all the RCPT TO's piggybacked *when there's actual mail*. So they'd have to work a lot harder under your scheme. POPing once (one list mailing) versus processing one email with zillion RCPT TOs (one list mailing) is not a very big cost difference. One might be slightly less than the other and we really can't say which one, but it is irrelevant because the difference is insignificant. Actually it is more likely that when they POP they will get several messages at once, so less cost than catch several SMTP emails. Also they know a priori the correlation of receivers to POP, which can be optimized with time, versus having to build a new mapping table in real-time every time they process an SMTP with RCPT TO. And let's *THINK* for a moment here - what is your proposal *REALLY* going to change? We already have many estimates that 50% or so of all e-mail is spam. Let's take that as a given, and let's make the rash assumption that the rest is 25% mailing list traffic and 25% person-to-person. It be more interesting to know what the real stats are on the other 50%, because I doubt that 25% is legitimate bulk email. It seems that you live in a different (mailing list centric) world than I and most normal people live in. I join mailing lists for a short time to get something done, then I leave asap. Most of the people I know and the many thousands of customers I come into contact with, seem to not even know how to use a mailing list. With 500 million people on the internet, I would venture that 80% don't even know what a mailing list is. They may use Yahoo Personals, and not even realize it is a mailing list. Since the email is being directly deposited into the Yahoo account, they have no clue. Any way, let's follow your line of debate... So what you want to do is take the 25% of the list traffic that works just fine on the current infrastructure, No it doesn't work fine. My gf complained that she couldn't find her Yahoo Personals email amongst the 500 spams she gets per day, of course that makes me happy but that is besides the point :) and is usually quite easily whitelistable via a number of different methods - Whitelisting can be subverted by spammers: http://www.cnn.com/2003/TECH/internet/09/01/spam.chainletter/index.html ...Herrick, however, admits that the practice could be a good way to bypass e-mail filters which block messages from senders who are not known to the recipient. Spammers could use chain letters to discover the addresses of people with whom you frequently communicate. Spam purporting to be from someone in your address book would sneak by filters. If I were a spammer, I'd be working very hard to perfect this technique, he said... and move it to something totally different. And what you're left with is a 2-1 mix of spam and personal mail that you yourself admit things like the DCC and spam filters are unable to perfectly distinguish. The whole point of the change is to enable elimination of the spam which can not currently be done. See my response to John C Klensin, regarding chicken and egg and the example benefits to
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 7 Sep 2003, Shelby Moore wrote:
I run a few mail servers, and have built many more. I personally would
have no desire for my mail to be handled by POP3, passed in cleartext
across the public internet, when I simply log into
my machine securely (locally or remotely) and type mail to access my
email.
There is nothing in my proposal that requires your private email to deviate from
your current more secure methods of SMTP delivery.
Shelby, by doing things this way i get no bulk email. Noone in their
right mind would target a server with 15 users for spamming... this
happens to mail providers with millions of subscribers.
By definition, any thing that is bulk is no longer private, because you can't
control what another person will do with the same information.
Why don't we try to eliminate it all in one fell swoop... bulk email, junk
postal mail, telemarketing calls, the whole lot of it. There is an
annoying company called the social security something or other who for
years sent me certified letter after certified letter saying that I should
have one of their numbers, and please contact them.
Further, I am not interested in having my mail sit on someone elses
server.
Your email would not. Only bulk email which is shared with many other people.
Good, because I don't want any bulk mail on my server. You can collect
all you want for me on some random server though. Not interested in
wasting either the cpu cycles or disk space it would take to deal with it.
Don't get me wrong Shelby, I HAVE a POP3 server, that I built
myself, for the use of friends and affiliates. Nor do I have the desire
to collect 2-3000 mail boxes in my lifetime, for all of the business
relationships I may make during this period.
It is a wrong assumption to equate business email with bulk email.
---
#!/bin/sh
rant(){
echo (
I am of the opinion that I never want to be sold anything in my life. If
I want it, I can find it.
These people deploying spammers as marketing agents are no different than,
or possibly are, the people who deployed call center sweatshops as
marketing tools, and bulk mailers eating thousands of trees proclaiming
what great deals they have on cars this week. The truth is, if someone
wants to spam bad enough, they will. We could rewrite smtp 5 times, and
it would be engineered, reverse-engineered, reengineered, implemented,
modified, patched, and kludged by thousands of people for various reasons
each time. And every time one of those reasons would be spam. As long as
we live in a profit-driven society anyway. Its too bad we can't just
hold hands and sing cumbyah, but it happens that way sometimes. This has
been rehashed time and time again on this list, unfortunately, which is far
from the venue for this discussion.
Who volunteers his mail server for the spam-discussion-redirect list that
will effectivly be the /dev/null for spam related topics that arrive at
the ietf? For those of you unfamiliar with UNIX or any of its
derivatives, /dev/null is a storage device for important internal system
processes. What say you IETF? Can we engineer a solution to this nasty
problem?
) | /bin/mail -s spam [EMAIL PROTECTED]
sleep 12
rant
}
rant
---
It is a wrong assumption to equate commercial email with bulk email.
rant
Seems easier than a complete rewrite
and redeployment of SMTP, which is what would be necessary inthis
instance.
Don't get me wrong, but respectfully, this has *NOTHING* to do with SMTP. SMTP is
not involved and not changed.
Therin lies the flaw in your plan, as smtp must continually change in a
distributed fashion in order to effectively reduce the amount of
egregiously time-wasting email that flows through its veins.
Shelby Moore
http://AntiViotic.com
sleekfreak pirate broadcast
world tour 2002-3
live from the pirate hideout
http://sleekfreak.ath.cx:81/
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
From: Shelby Moore [EMAIL PROTECTED] ... And I tell *MY* UIDL from Keith Moore's UIDL from Vernon Schreyer's UIDL how? How did I get involved in this? 15 years ago I had a boss that finally taught me to never use real names in examples or scenarios even when I was sure I was being nice. It's better to invent names, and to take care that they're far from anything real. You must assume that someone will take you literally in bad ways you'll never predict. ... Ironically you mention Vernon (is that a hint?), because if I remember correctly he runs the DCC and recently had a similar unexplained overtly negative reaction recently when I tried to contact him regarding some improvements to DCC. I suppose it is possible that your negative responses here may be explained by similar vested interest with him. Maybe, but in fairness I won't immediately assume the worst ethics of you. Worst ethics? Mr. Moore contacted to ask me to sign a non-disclosure agreement so that I might learn how to make the DCC effective based on his new intellectual property. He was also interested in buying copies of DCC data. My responses started cool and eventually became as overtly negative as I could without calling him names. I told him that he had disclosed nothing of his intellectual property to me, that there is no plausible chance we might ever do any business, that I wished he would stop sending me mail, that I did not (and do not) want to hear about his intellectual property, and that his lawyers should be aware that my document retention policy for email is forever in a bank vault and that I've been known to take formal notes during telephone calls. Perhaps Mr. Moore's harping on his intellectual property and an NDA sent me into a raving paranoid break, but many of us know real life stories that started similarly and turned out poorly from some points of view. He sent a few more messages after my request that he stop, but eventually quit. If this triggers more, I'll file them with the others, not in file 13 but in that bank vault. Media's cheap today. I'm writting this only to urge caution. Some people are innocent and well meaning and only appear otherwise to us paranoids (split personality and delusions of royal or editorial grandeur as well). Other people really are dangerous. A few years dealing with spammers or with patents and intellectual property experts should make anyone spooky. I realize the most dangerous people don't seem dangerous. Maybe that proves Mr. Moore's virtue, or not. BTW, in case you think my anti-spam business depends on this proposal, that would be false assumption. Actually one of the key advantages of my anti-spam algorithm (the automatic white listing of legitimate bulk email) would be rendered unnecessary if this proposal was put into effect. I hope I don't need to go read about that automatic white listing and so forth. I find Mr. Moore's technical writing as inpenetrable and painful to read as my own. For now I'll assume that the IETF archives provide sufficient protection. ... So you are indeed heavily involved with the DCC! Yes I know that is a weakness in the way the DCC measures bulk, but that is irrelevant to the point above. Please notice my resolute failure to ask about that weakness. I've no clue what Mr. Moor's intellecutal property is, except that I suspect it is similar to some ideas related to counting mail from IP addresses mentioned by others in public...and conveniently archived by Google. If I did not realize it was happening, then I would not have created http://AntiViotic.com . I'd be using the DCC instead. I also wouldn't have approached Vernon about improving the DCC (only to be turned away in nasty tone). That's me, the ol' evil, nasty, non-team playing, selfish, negative, unfair, anti-comerz, kook himself. Another thing that Mr. Moore convinced me during our exchanges is that his notions about the nature of the DCC works differ significantly from mine. There's nothing wrong with that, since without Mr. Moore's intellectual property, the DCC is not effective or whatever. I hope the fact that every time I turn around I bump into another person demanding attention for the Ultimate Wonderful Final Perfect Solution to The Spam Problem is a good sign instead of a perverted maturation of the anti-spam industry. Vernon Schryver[EMAIL PROTECTED]
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
The second is raising the cost to the spammer. Personally, I like the idea of taking up a collection among the ISPs and other providers, and hiring some good ethnic muscle (there's competition in the field, a number of experienced and ruthless groups are available). I'm sure the spam problem would change drastically if the spammer was seriously having to balance the mentioned $300K for bogus enhancement pills against having their kneecaps broken by one group or worse by one of the other groups... What, who needs a kneecap job done? Contact me off-list;) We have very reasonable rates wait. sorry. that was spam.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 01:43 AM 9/7/2003 -0400, you wrote: On Sun, 07 Sep 2003 13:07:10 +0800, Shelby Moore said: It is a wrong assumption to equate commercial email with bulk email. Which is why you're trying to rewrite how bulk email is done in order to deal with *one segment* of commercial e-mail. Now I understand fully. No I am trying to eliminate (drastically reduce) spam (UBE). And, I am trying to give receivers the ability to opt-in and opt-out of all legitimate bulk email under their own power. In my mind, that is more important than existing legitimate bulk email paradigm. I can understand you and other's vested interest in existing legitimate bulk email paradigm, but I don't agree it is more important. When spam is 99% of all email you recieve, you may begin to care also. It won't be too long from now... Fortunately what this discussion is proving to me so far, is that architectual change will be resisted fiercely by vested groups that control the design of the internet. This means that my new (soon to be patent pending) algorithm for http://AntiViotic.com will have a near monopoly in the market. So for me and users of my service it will be great. But for the rest of users, spammers will be forced to actually increase the amount of email they send in the self-feeding model of my algorithm. Unfortunately, the rest of the email universe will be imploded by an accelerating rate of spam. Just imagine what spammers will do when confronted with an insubvertable algorithm that can detect bulk. I came here to see if there was a better way, but I guess we will have to do it the hard way (and very profitable for me). This is all conjecture (vaporware) at this point...but very well researched conjecture. I'll be back here in this list later (probably a year from now) when your needs have changed to a more dire state regarding email.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 01:54 AM 9/7/2003 -0400, you wrote: The evidence indicates that the senders will use whatever is more likely to result in the receiver seeing the message. This is different from seeing it where the receiver would like to see it. I get your point and it is a reasonable one that must be taken seriously. However, I think that at some point soon, users will have no choice but to either stop using email (because it doesn't work efficiently any more) or look for a solution to spam. At that point, I think they will demand change (already seeing early warning signs of that now). Also you are assuming that legitimate bulk email is very important to most receivers wherein I think for most spam is a bigger issue. Lastly, I don't think my proposal has to be any more inconvenient than existing sub/unsubscription procedures, which require trading several emails, versus inputting one POP domain in an email client. I understand the concept of diminishing utility in economics, so I take your point seriously. And I will ponder it more. The point of the usenet reference analogy is that if placing things where people have to go look was sufficient, folks would not have started using spam email. But there is a key difference. Users started preferring email without the agreement that it would have spam rate increase from 8% to 50% in 2 years. When the noise gets too high, you turn off the TV and do something more productive (remember antenae reception?). If you have a blackout every 5 minutes, you giveup trying to do anything with appliances. Etc.. They would have used other techniques. And they will if spam continues to increase. They already are. And those techniques may have much worse effects than a correct architectual definition of spam. For example, legitimate mailing list traffic is getting falsely classified as spam. Maybe not for you, perhaps because you are expert, but I assert your experience is not likely shared equally by the 500 million... Many (most) of the spammers are simply interested in any technique that will get the message in front of the recipient. Hence, they will use the path that is checked most often (the personal mailbox). Yes my proposal depends on that fact. Once you have the legitimate email separated from the spam architecturally, then you can effectively increase the cost of spamming to the point it is a non-economically viable activity. Remember the economics of spam is precisely what makes it so noisy. If spam ever got like 5% response rate, then it would not be such a big problem as it is. The fact that is averages something like 0.005% response rate is what makes it both a big noise problem and also makes it very fragile cost wise. The typical individual spammer does not make $300,000 a day. More like less than $100,000 a year. And the institutionalized spammers (the economy-of-scale bundlers) would make nice big fish for FTC. The more you can concentrate, the better. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Don't get me wrong, but respectfully, this has *NOTHING* to do with SMTP. SMTP is not involved and not changed. Therin lies the flaw in your plan, as smtp must continually change in a distributed fashion in order to effectively reduce the amount of egregiously time-wasting email that flows through its veins. No you are counting the chickens before the eggs. If you have no more eggs, then you won't get any more chickens. The misuse of SMTP has to be attacked at the root cause, which is my proposal (go study all my posts for full logic), rather than morphing SMTP in a never ending game of cat and mouse.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 14:02:30 +0800, Shelby Moore said: POPing once (one list mailing) versus processing one email with zillion RCPT TOs (one list mailing) is not a very big cost difference. One might be slightly less than the other and we really can't say which one, but it is irrelevant because the difference is insignificant. When you compare a successful SMTP to a successful POP, yes. Actually it is more likely that when they POP they will get several messages at once, so less cost than catch several SMTP emails. Actually, the most likely is POP once an hour for a once-a-week posting, and you've totally blown 167 pointless transactions. And in fact, unless you're able to make the POP check frequency less than the posting frequency, you'll lose. Whitelisting can be subverted by spammers: http://www.cnn.com/2003/TECH/internet/09/01/spam.chainletter/index.html If I were a spammer, I'd be working very hard to perfect this technique, he said... And your proposal does nothing to stop this, since it won't look like bulk mail, it will look like personal mail. From people you know and trust, and everything. Yes we probably do. Just because the DCC can not measure bulk email reliably doesn't mean Hosts, ISPs, and other software can not. BrightMail already is ( just signup for an Earthlink account and try really hard to get some spam), and I will also be probably be demonstrating something soon. So why do we need to move off mailing lists, when the problem is solved? It is has nothing to do with what spammers will or will not do. It has to do Actually, it has everything to do with what they will or wont do. with what Hosts, ISPs, etc are currently prevented from doing. Since they can not determine what is spam, they can not enforce any law. And separating out mailing list traffic doesn't change matters, really. You're left with a lot of non-bulk high-volume business e-mail (yes, you WANT this to get through, otherwise Amazon doesn't have a way to tell you easily about a problem with your order, or similar), a lot of person-to-person mail, and a lot of spam pretending to be one or the other of the above. The only thing you've cut out is spam to mailing lists - and if you can solve the OTHER two flavors above, then this third is a non-issue anyhow. Given that so much spam is already breaking some law, why do you think the ISPs could enforce a NEW law would make any difference? From another note: Yes my proposal depends on that fact. Once you have the legitimate email separated from the spam architecturally, then you can effectively increase the cost of spamming to the point it is a non-economically viable activity. This would be wonderful, except what you're separating isn't legitimate/spam, it's mailing list/non mailing list. The two are orthogonal concepts. *plonk* Somebody wake me up if Shelby starts addressing the actual problem, rather than an orthogonal non-problem pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 12:56:04 +0800 Shelby Moore [EMAIL PROTECTED] wrote: What you are saying IMO, is that you can't force bulk emailers or spammers to use opt-in. Let's be even clearer. What's being claimed is that you can't force bulk emailers to send their email via pull technology (whether this means providing their own POP servers, IMAP servers, NNTP servers, web servers, whatever) while everyone else can still use push. And the question isn't really whether bulk mail can be statistically distinguished from non-bulk mail (since that's really just a matter of whether you can get people to adopt a definition of bulk in terms of externally visible traffic properties) - the question is whether you can enforce that rule. IMHO - most recipients don't want to get their mail that way (and many of the deployed user agents don't support it), most senders don't want the increased burden of providing POP/IMAP/NNTP/web servers and the necessary customer support, and there are enough ISPs that derive significant revenue from selling bandwidth to spammers that it would be extremely difficult to get them all to enforce this. In short: nobody has sufficient incentive to adopt this. Of course there's nothing wrong with defining another way to distribute bulk mail that people can use if they wish. If it works well, some people will use it. The stretch is to insist that everybody do it this way.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I'll be back here in this list later (probably a year from now) when your needs have changed to a more dire state regarding email. Thank you for playing. sleekfreak pirate broadcast world tour 2002-3 live from the pirate hideout http://sleekfreak.ath.cx:81/
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Information theory says that such things are impossible. One can not construct a spam-free protocol because this is the same problem as constructing a system free of covert channels, which information theory says is impossible. It is not simply hard. It is impossible, like perpetual motion. After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. I have recently discussed the issue at length with some PhD's and other very smart people from one of my old companies, and we determined after a great deal of discussion that covert channels (and similar concepts like side channels and information leakage, and other terms used in information theory literature) don't require cooperation as a necessary condition. The term Covert Channel is mostly used in liturature analyzing operating systems. However, the general concept is referred to in non-OS liturature as Side Channels or Information Leakage, or Sneaky Channels. The concepts are essentially the same. Indeed spam can be modeled as a communication channel that is multiplexed with email, which is itself multiplexed with TCP, which is multiplexed with IP, which may be multiplexed over a variety of physical media. It was also realized (towards the end) that spam is in fact a cooperative covert channel, since the reader cooperates when the receive the spam and open it for reading. Only _after_ reading the spam message are some people offended, but this is of no consequence to the application of information theory because the communication has already taken place. All they can do, having now detected the abuse, is try to suppress further abuse that is similar. But this is a simpler way to think about it: Spammers can continue to claim they are legitimate emailers, because they _ARE_ legitimate, so far as we know before they send email. And even so far as we know _before_ someone _READS_ their email. Only after reading their email, and perhaps only after some investigation, can we know for sure that the sender and message is conducting abuse or in violation of their AUP. Some might ask about blocked abuse. Subsequent abuse is blocked because it is similar to previous abuse (by IP address with blacklist), or by content with content filters. We can anticipate some likely abusive content, but cannot identify all abusive content in advance. However, I looked at your proposal, and it appears that you are trying to create a pull mechanism rather than a push mechanism for message delivery . This paradigm has already been implemented as Usenet News. It is not immune to spam, though it distributes spam and other broadcast messages much more efficiently than typical email systems. However, bandwidth consumed by spam email (indeed bandwidth consumed by _ALL_ email combined), is still minor, so we are really not in need of a more efficient means of distributing spam, or email for that matter. --Dean On Sat, 6 Sep 2003, Shelby Moore wrote: Request for opinions on whether to creating a working group or publish the following idea as an internet draft? Spam is big problem that is getting worse. BrightMail.com (which claims to process 10% of world's email) claims that the percentage of spam out of all email has grown from 16% in Jan. 2002 to 50% in Aug. 2003. A fundamental unsolved problem of doing any thing about spam, is there is currently no unambiguous definition of spam as an enforceable internet standard. This has been architectually impossible to define because the receiver is the subjective determinant of which bulk email is solicited and which is spam (UBE). ISPs, Hosts, legislators, judiciaries, and even anti-spam software, have a fundamental problem in that definition of spam as UBE is currently architectually unenforceble due the fact that subjective determination of unsolicited current happens after the email has been delivered to the receiver. My idea is to create an internet draft, RFC, and hopefully internet standard, that would define a simple architectual paradigm for legitimate bulk email that unambiguously separates it from spam (UBE). Simply define that legitimate bulk distribution of email should be done by mechanism of each bulk distributor providing a public POP3 (and IMAP) account or server, rather than sending the email directly. In the case of a public distribution (e.g. most direct email and mailing lists), a POP3 (and IMAP) account of user anonymous with password none would suffice. In the case of private dissemination (private mailing lists), a POP3 (and IMAP) server with individual accounts could be provided. The elegance of this paradigm is that users then control the opt-in/opt-out database, by configuring their email client to POP email from only the bulk POP accounts they wish to subscribe to. The effort to support this paradigm is minimal because it uses existing email
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On zondag, sep 7, 2003, at 21:45 Europe/Amsterdam, Dean Anderson wrote: Information theory says that such things are impossible. One can not construct a spam-free protocol because this is the same problem as constructing a system free of covert channels, which information theory says is impossible. Nobody cares. Making a roof 100.00% impervious to water molecules may be impossible, but that doesn't mean we have to resign to getting wet every time it rains. It is not simply hard. It is impossible, like perpetual motion. So when exactly was the earth supposed to stop moving? After I first posted this on IETF a while back, someone suggested that covert channels require cooperation, and that spam therefore isn't a covert channel. Where does this covert channel stuff come from anyway? But this is a simpler way to think about it: Spammers can continue to claim they are legitimate emailers, because they _ARE_ legitimate, so far as we know before they send email. And even so far as we know _before_ someone _READS_ their email. Only after reading their email, and perhaps only after some investigation, can we know for sure that the sender and message is conducting abuse or in violation of their AUP. This goes for each individual message, but the spammer's achilles heel is that they need to send out incredible amounts of email in order to fulfill their objectives, whichever those are. Detecting bulk mail is doable, and it shouldn't be too hard to come up with something to differentiate legitimate bulk emailing from spam. For instance, we can reverse the burden of proof here and only allow know bulk emailers. However, I looked at your proposal, and it appears that you are trying to create a pull mechanism rather than a push mechanism for message delivery . This paradigm has already been implemented as Usenet News. My point exactly except that usenet doesn't have to be significantly more pull than email (where you need your client to check for new mail as well). It is not immune to spam, though it distributes spam and other broadcast messages much more efficiently than typical email systems. Ouch! :-) Fixable with authentication.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Iljitsch van Beijnum [EMAIL PROTECTED] wrote: It is not immune to spam, though it distributes spam and other broadcast messages much more efficiently than typical email systems. Ouch! :-) Fixable with authentication. no. --Johnny
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 04:24 AM 9/8/2003 +0800, Shelby Moore wrote: At 11:51 AM 9/7/2003 +0800, you wrote: You can get mail no matter where you are with a POP account also. shelby, that's actually not true. If you have an enterprise email service that requires access to a VPN and the internet service you access it with (e.g hotel room ethernet) has a bad firewall configuration, you may never get to the mail. I speak with personal experience - the hotel I am in right now has screwed up its firewall. I ended up having to find an 802.11 hotspot to get to my email. I understand but that was not my point. My point is that you can put a web-based interface on top of your POP account to access it any where. You still have a POP account which you are accessing any where if that is what you want. The web-based interface is just another form of an email client. that's different - what you said was as quoted above. I agree that if you design the web server properly, you can use a web interface, but you run the risk that with this design, you may never be able to pull the email later, POPStyle, into your computer. Although it is theoretically possible, using POP (rather than IMAP) to leave the mail on the server until you pull it again with POP, many servers appear to clear out the mail after POPing it. I think John Klensin made that observation in an earlier exchange. The point is that you don't need to use a web-based email without an underlying POP account in order to access email from any where. There are even places where HTTP web-based interface won't work (e.g. cell phone) and so you need to use a different form of email client to access. Still you can have an underlying POP account that mail is being drawn from. see above. v Vint Cerf SVP Technology Strategy MCI 22001 Loudoun County Parkway, F2-4115 Ashburn, VA 20147 703 886 1690 (v806 1690) 703 886 0047 fax [EMAIL PROTECTED] www.mci.com/cerfsup
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Keith, IMHO you started an excellent line for further debate (and not just because we have the same last name :). It would be nice to see debate from both sides so that pros and cons could be fully explored. I am not sure I am the one to carry the debate to extreme end (due to time constraints), but I will initiate the counter argument. I came here to make a proposal. It is not a proposal I would champion to the end game alone. The feedback is valuable (hopefully to community-at-large as well), whether the proposal florishes or not. What you are saying IMO, is that you can't force bulk emailers or spammers to use opt-in. Let's be even clearer. What's being claimed is that you can't force bulk emailers to send their email via pull technology (whether this means providing their own POP servers, IMAP servers, NNTP servers, web servers, whatever) while everyone else can still use push. Enforcers will force the adoption. Enforcers being Hosts, ISPs, legislatures, judiciaries, software, etc.. Once you define that *legitimate* bulk email is only delivered by pull, then enforcers can start to filter bulk email. This will force everyone to get their *legitimate* bulk email via pull. Else they will not get it. If you have a choice between receiving your *legitimate* bulk email via pull, or not receiving it, what are you going to do? Now below you argue that enforcers will not enforce. To me that is more viable argument, one that needs to be debated. However, what is the harm in making an RFC and then find out if enforcers will enforce?? more below... And the question isn't really whether bulk mail can be statistically distinguished from non-bulk mail (since that's really just a matter of whether you can get people to adopt a definition of bulk in terms of externally visible traffic properties) - the question is whether you can enforce that rule. Agreed. Good point about benefits (real results) of agreeing on definitions. That is in essence why I am making this proposal. I am proposing we make a definition of spam which is both agreed upon and not ambiguous to enforcers. Simple as that. In my mind, the thing to consider is what is the cost of making this definition compared to what is the benefit?? To me, that is one of the major aspects that need to be debated. Economics (in the larger sense of economy as efficiency or survival/evolution, not just financial systems) *always* rules the outcome. IMHO - most recipients don't want to get their mail that way (and many of the deployed user agents don't support it), Most people already get their email through POP, even if there is a web-based client on top of it. We already went through the discussion that Hotmail and Yahoo have the ability to access POP accounts. They wouldn't have the feature if people aren't using it. most senders don't want the increased burden of providing POP/IMAP/NNTP/web servers Only *legitimate* bulk email senders (mostly that is mailing lists) would have to do this. Non-bulk email and spam does not change it's paradigm. Spam is dealt with by enforcers. Only *legitimate* bulk email senders (mostly that is mailing lists) would have to do this. Only *legitimate* bulk email senders (mostly that is mailing lists) would have to do this. Just making that very clear. *Legitimate* bulk email senders already have a burden of maintaining mailing list databases, maintaining a subscribe and unsubscribe support, handling bounces, etc...s and the necessary customer support, I assert there will be less or similar hassles with supporting a pull paradigm for *legitimate* bulk email than all the hasslings of teaching newbies how to subscribe and correctly interface with Majordomo mailing list. I do not think it is a big difference. And if you are protecting the mailing list administrators to save spam, then that seems like a bad set of priorities for internet email as a whole. and there are enough ISPs that derive significant revenue from selling bandwidth to spammers that it would be extremely difficult to get them all to enforce this. This is wrong. This would be saying that the economy of spam is significant compared to the internet economy as a whole. ISPs are losing $ big time to spam. In short: nobody has sufficient incentive to adopt this. Maybe nobody in this list, but let this thread sit in Google for a while and see what might snowball over time :) Of course there's nothing wrong with defining another way to distribute bulk mail that people can use if they wish. Great. That is what I said above. Why not see what the enforcers will do? They can not do any thing now, because their hands are tied to ambiguous definition of spam as UBE. They need spam == *BE in order to act. If it works well, some people will use it. The stretch is to insist that everybody do it this way. No. If it works well (if the enforcers adopt), then everyone
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
This is getting way off topic. One of the other things you see to be handwaving a bit about is the notion of handing out user IDs, passwords, and other credentials to mail accounts to people so they can help with spam (or other problems). My proposal has nothing to do with IDs so let's just drop that line of discussion or respond with a different subject to move it to a different thread. Sure, I can find a web-based something-or-other to access my POP3 mailbox (if I had one). Agreed. the web site listed in your signature line seems to mostly run people around in circles My web site and businesses have nothing to do with this proposal. The proposal is to define that *legitimate* bulk email must be sent and received by pull instead of by push. So that all remaining bulk email is spam. This changes spam from Spam == UBE (an ambiguous definition for enforcers) to Spam == *BE (UNambiguous definition for enforcers). How *BE gets dealt with by enforcers is not related to my businesses or web site. Enforcers is a widespread term meaning any one who will enforce the proposed RFC against spam. FYI: (completely off topic so please don't respond to list on this point) The reason the web site runs in circles is it isn't finished yet, nor ready to be released to public at large yet. It should be completed in a few more days. Shelby Moore http://AntiViotic
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Iljitsch van Beijnum [EMAIL PROTECTED] wrote: On maandag, sep 8, 2003, at 00:08 Europe/Amsterdam, Johnny Eriksson wrote: It is not immune to spam Fixable with authentication. no. As each individual news article is piped through a relatively small number of servers in the core of the distribution system, it becomes relatively easy to blacklist known offenders. That is, if they are recognizable as such. No way. If this was indeed the case, it would already have been handled by blacklisting known AS numbers / prefixes in the routing system. This is not the case. See below. This is where the authentication comes in. This is where layer eight and above comes in. There are enough people that profits from this that us low-lifes that want spam to stop just don't count if you want to play by the rules. I'm all for the kneecapping proposal by citizen V.K. --Johnny
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
As each individual news article is piped through a relatively small number of servers in the core of the distribution system, it becomes relatively easy to blacklist known offenders. That is, if they are recognizable as such. This is where the authentication comes in. The tricky part is optimizing the time/difficulty it takes to blacklist vs that of obtaining a new identity. Even though this is a good point, I'd prefer to stay off the discussion of authentication, because the proposal need not depend on it, as you point out below... Also note that with usenet news, unlike email, it is possible to remove spam that has entered the system, limiting the audience that sees the message and thus the effectiveness of spamming. Also bear in mind my point that the whole point of moving legitimate bulk messaging to pull is so that spam can be dealt with unambiguously by enforcers on bulk email. So spam (all remaining *BE) gets reduced by the paradigm switch to pull for legitimate bulk messaging. Since I assume mailing lists would still use email as incoming source primarily, then incoming spam is reduced. You could certainly switch to a authenticated interface (e.g. https web page) for incoming mailing list traffic, but I think that is unnecessary to my proposal. Also remember that some portion of legitimate bulk messaging is legitimate (meaning that receivers want it) corporate bulk mailings. They would also be forced to the pull paradigm, but not their normal single messaging, such as the receipt for what you buy on Amazon, only their bulk email. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
As each individual news article is piped through a relatively small number of servers in the core of the distribution system, it becomes relatively easy to blacklist known offenders. That is, if they are recognizable as such. No way. My proposal does not depend on authentication of what is incoming to the pull servers, because by definition it enables the enforcers to filter *BE (all bulk email) which (by the proposal) is defined to be all spam. This is a non-issue relative to my proposal. If you are really paranoid, use https web page to take incoming from authenticated users. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Excuse me, it is a valid issue that spammers will try to pipe through mailing lists (legitimate bulk email) to avoid *BE enforcers. Mailing list administrators will continue to carry this burden and probably more so under my proposal. Thus yes I agree that authentication of incoming to pull servers is a major consideration. However unlike with current paradigm, the receiver can then opt-out of the spam. Thus darwinism will force effective solutions, because those mailing lists that are successful will have more receivers than those who are not (because the receivers how power of opt-out of spammy lists unlike now where they can not opt-out of spam email because their legit bulk email not unbundled). And remember that commercial legitimate bulk email is not effected by incoming pollution issue. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
However, what is the harm in making an RFC and then find out if enforcers will enforce?? you appear to presume that you can get consensus support for such a plan from within IETF. even if you could get such support (which you cannot) note that there's no enforcement of IETF's other opinions, even in cases where failure to adhere to IETF standards costs billions of dollars every year. why would this case be any different? there are a lot of ways to solve the spam problem that will work if everybody does it my way. so far, nobody has figured out how to impose their will on the rest of the net. Keith
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
so far, nobody has figured out how to impose their will on the rest of the net. thankfully Keith sleekfreak pirate broadcast world tour 2002-3 live from the pirate hideout http://sleekfreak.ath.cx:81/
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I am nearing the end of my allow time to respond, so if I do not respond in future, it doesn't mean I agree :) below... 2. Regarding additional burden on *legitimate* bulk message *senders*: a. These senders are much, much fewer than the # of receivers suffering from spam. Any incremental cost on the few is justified. yeah, well good luck convincing *them* of that. especially when you can't even convince *us* of that. You (us not including me and perhaps some others) are apparently them, so I see no need for the word especially. Further, there is no need to convince the minority. Majority wins. If enforcers successfully block all bulk email with my proposal (as an internet STD) sanctioning it, then receivers (the minority) will dictate the outcome. I assert that even if 50% spam is not bad enough to make the 500 million rise up, just wait until it is 90+%, which again I assert won't be too long from now. b. The senders are already quite resource savvy, else they would not be sending *legitimate* bulk (in statistical significance) messages. I doubt the incremental cost is significant to cause failure. let's see. it doesn't cost the spammers much to send bulk mail... Oh it costs spammer something. They are definitely more resource savvy than the majority, which was my point. The burden on the minority is less important, if the majority is making big gains from the proposal. so why do you assume that the costs are significant to legitimate senders? They have to maintain mailing list servers, mailing list databases, manage support for subscribers, handle incoming spam, etc.. All things which an average (the majority) individual user of email doesn't have to do (or won't bother to do). c. And compare this burden to the burden they have with dealing with spam, which would be eliminated by this proposal. If spam isn't eliminated, then they need not adopt the proposal. most of the burden of dealing with spam is on the recipient, not the sender. Agreed currently. But not once my proposal succeeded, which is a correction I made after making this point #c above. Point #c is invalid. Please remove it. apparently you think that legitimate senders should pay for the cost of lessening the burden on recipients from illegitimate senders. Insert bulk after legitimate and illegitimate, then I agree with your statement. The reason is because *legitimate* bulk message senders are going to pay one way or the other soon. It is like the old Fram oil filter commercial in USA, Either pay me now (replace filter) or pay me later (replace motor) As the rate of spam approaches 90% or 99%, receivers are going to opt to block all bulk email, irregardless of whether my proposal is in place. They won't have any other choice, else quit using email. Now that is profound prediction! why do I think you're not likely to get much support for that view from the senders? Oh yes to be expected that human nature tends to manicure (defend) their own feet instead of watching where they are going and what they will soon bump into. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
However, what is the harm in making an RFC and then find out if enforcers will enforce?? you appear to presume that you can get consensus support for such a plan from within IETF. No, no. I try to never beg. I came here to make a public proposal and some points for the purposes of public record. I never expected any one to really do any thing on this proposal now. But by placing the idea into public domain now, as the rate of spam approaches the asymptote of 100%, then mailing list administrators can come back to this idea as a way to save themselves. Actually at that point, they won't need the RFC or STD, because all bulk email will be blocked and they will be forced to just go straight to offering a pull solution (whether it be web, usenet, pop or whatever) so recievers can still get their mailing list messages amongst the 10,000 spams per day. I am not sure exactly how long this will take, but without another solution or paradigm shift, I figure 2 - 5 years tops. even if you could get such support (which you cannot) How do survey their opinion?? note that there's no enforcement of IETF's other opinions, even in cases where failure to adhere to IETF standards costs billions of dollars every year. why would this case be any different? Because enforcers want to do something and will be under increasing pressure to do something against bulk email. there are a lot of ways to solve the spam problem that will work if everybody does it my way. I have seen none. Email signing won't work. No amount of !plonks from Randy Bush will change that. Changing SMTP won't work long term, etc.. so far, nobody has figured out how to impose their will on the rest of the net. It is not my, your, or IETF will, but the votes of 500 million (through their actions against spam) that will impose. Shelby Moore http://AntiViotic.com
Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Request for opinions on whether to creating a working group or publish the following idea as an internet draft? Spam is big problem that is getting worse. BrightMail.com (which claims to process 10% of world's email) claims that the percentage of spam out of all email has grown from 16% in Jan. 2002 to 50% in Aug. 2003. A fundamental unsolved problem of doing any thing about spam, is there is currently no unambiguous definition of spam as an enforceable internet standard. This has been architectually impossible to define because the receiver is the subjective determinant of which bulk email is solicited and which is spam (UBE). ISPs, Hosts, legislators, judiciaries, and even anti-spam software, have a fundamental problem in that definition of spam as UBE is currently architectually unenforceble due the fact that subjective determination of unsolicited current happens after the email has been delivered to the receiver. My idea is to create an internet draft, RFC, and hopefully internet standard, that would define a simple architectual paradigm for legitimate bulk email that unambiguously separates it from spam (UBE). Simply define that legitimate bulk distribution of email should be done by mechanism of each bulk distributor providing a public POP3 (and IMAP) account or server, rather than sending the email directly. In the case of a public distribution (e.g. most direct email and mailing lists), a POP3 (and IMAP) account of user anonymous with password none would suffice. In the case of private dissemination (private mailing lists), a POP3 (and IMAP) server with individual accounts could be provided. The elegance of this paradigm is that users then control the opt-in/opt-out database, by configuring their email client to POP email from only the bulk POP accounts they wish to subscribe to. The effort to support this paradigm is minimal because it uses existing email paradigm. Legitimate bulk senders have to change from a broadcast (push) metaphor (e.g. Majordomo) to a pull metaphor simply by depositing their outgoing email in the public POP account they create. Receivers simply follow instructions to POP bulk email they want, instead of the equally complex task of subscribing to bulk email. This accomplishes several goals: 1. Any bulk email is then spam (receiver has not opted in) and can be dealt with by ISPs, Hosts, legislators, judiciaries, and anti-spam software. 2. Receivers now have uniform control over opt-in/opt-out policy without a global authority 3. Legitimate bulk senders can be insured that they or their email won't be misclassified as spam 4. Those who send UBE can no longer claim they are legitimate or that receiver has opted-in (ambiguity removed) and can be dealt with by ISPs, Hosts, legislators, judiciaries, and anti-spam software. 5. With a pull paradigm, the load (resource usage) on the public internet, sender, and receiver is reduced, because I venture that a majority of bulk email sent would not be pulled. I think this paradigm would empower Hosts, ISPs, legislatures, and judiciaries to do more about spam (incoming) and spammers (outgoing), because their hands would not longer be bound by ambiquity. I realize that some vested interests, such as direct emailers or those invested in push based mailing lists, might resist. However, I think the benefits outweigh the limited costs to migrate. Some direct emailers might resist because some may prefer being able to cloak spam under the guise of solicited. Legitimate bulk emailers stand to gain a lot by separating themselves from the noise of UBE. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
One additional point in response to this point: So let's see.. Currently, if your bank sells your e-mail address to another company, you get spammed. So instead, you'll have it so that you check your bank's POP server in case there's important mail about your mortgage. Seems like the obvious scheme is for the bank to charge the other company to put stuff in your POP mailbox. So you still get spammed... In addition to what I wrote below, remember that when the bank is sending me an individual email regarding some business I am conducting with them, they are not sending that email in bulk (to any one else). They could send that directly to my email. If they are indeed sending a similar email to ALL their clients at the same time, then in that case they are sending bulk email. So with my proposal, it just forces business to separate their business email from their marketing bulk email. If I trust my bank to send only important bulk email, then I can add that POP account to ones that my email client checks regularly (as regularly as I chose, not as my bank choses...gives me the control). Given that email is insecure transmission medium, no business should be sending me anything too important in email. They had better have an alternative means of getting in contact with me about important and urgent matters. ---I wrote before only No. Because you can chose to not check it and/or you at least know who is spamming you and can hold them responsible directly. Thus your bank would stop doing it, because they make $ by not losing your business.
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 09:58:47 +0800, Shelby Moore said: If it became an RFC or internet standard, and it became widely adopted, then it is reasonable to assume that email clients would add features to handle this . It is quite a low bandwidth operation (probably less than 1K bytes) to poll a POP server for email It's not the bandwidth - it's the fact that there are these annoying things called timeouts. For *each* server that isn't reachable, you get to wait a minute or so - suddenly those 150 checks are taking quite some time. And the queuing theory of 150 sites with intermittent connectivity doing a push to your POP server is different than what you see when you try to do 150 pulls But of course, if you actually *tried* this, you'd understand it... However, there is one key technological hurdle I did miss in my haste, there would need to be some mechanism so that the same user doesn't keep downloading the same messages over and over again. This would either require a special modification to the POP server and require each user to login with a unique user name. Hey.. what did I tell you? Everybody needs a login of their own... And as a side benefit, there would be no way for someone to subscribe me to a list without my permission, as can be done by sniffing an authentication email for Majordomo. If your confirmation mail is being sniffed, you have *BIGGER* issues. And if you have bigger issues, I suggest using the *proper* tools for the job. See RFCs 2362-2364 and 3156. If your issues are bigger than that, e-mail subscriptions are the least of your problems. False. You are correct that I missed this issue in my initial post. However, it need be only one POP account (one storage of emails) with flags for each user. In other words, the storage requirement need no increase drastically with number of subscribed users. Hmm... store each mail as an object with links for each recipient. A truly novel idea, our homegrown mail system implemented it back in 1992. The flags can either be stored at the POP server and then give each user a unique login id, Hey.. there's that unique login again. No only 6000 POP accounts. See above how email clients can handle the detection of new messages using UIDL. And you only need one anonymous login and no password (just configure the POP server to accept any login and password). And I tell *MY* UIDL from Keith Moore's UIDL from Vernon Schreyer's UIDL how? Have you actually *TRIED* to use more than 100 POP accounts under any current mail software? I will respond with similarly rhetorical question. Did you try to use Netscape 2 on most current web pages? Why make any application RFC if there can be no progress in applications? There's a difference - I'm not proposing a new scheme of doing things that involves a change to 500 million users. So I submit to you that if this idea is too hard to use with the current version of Outlook, it is a *non starter* as a practical matter. 50, even 5000, is not statistically bulk on internet scale. Is it not possible (or likely) to write laws without exclusions? Do you think Hosts, ISPs, and anti-spam software would not account for this statistical phenomenon? Only problem is that many spammers are *already* only dropping 40-50 copies of a note at a site at a time, specifically to work around that - then the rest of the spamming recipients at your site get a different version with a different From: line and a different source IP address. I submit to you that if you didn't realize this was happening, you may not be qualified to be suggesting proposals to counter it (Hint - if spammers weren't doing this, it would be trivial to block them, and we'd not be HAVING this discussion, right? ;) It's ironic that you're proposing this on a push-based mailing list provided by an organization that is probably not in a position to provide POP accounts for the 30,000 or so recipients of the the list. No. As I said above, they would only need to provide one POP account for this mailing list. And as I pointed out, you'll need to create 30,000, because one account doesn't allow you to keep track of who has already seen what messages. And no, you're *NOT* allowed to just say everybody can fetch all the UIDLs and we'll just tag them with the subscriber ID - go read and *UNDERSTAND* section 6.2 of RFC2298 in order to understand why. You might also want to go re-read the ASRG mailing list archives, your proposal (and variants thereof) has been kicked down the beach like a dead whale multiple times already. pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sat, 06 Sep 2003 23:07:44 EDT, [EMAIL PROTECTED] said: And as I pointed out, you'll need to create 30,000, because one account doesn't allow you to keep track of who has already seen what messages. And no, you're *NOT* allowed to just say everybody can fetch all the UIDLs and we'll just tag them with the subscriber ID - go read and *UNDERSTAND* section 6.2 of RFC2298 in order to understand why. Another reason why you need unique userid/logins for each subscriber - so that you can prevent forging a UIDL for somebody else to keep them from reading the message. Being able to do this (and if you have a shared userid, it's almost impossible to prevent) would make the Bernstein/Bush flamefest regarding list censorship look trivial in comparison... Oh... there's also this thing called webmail. Lots of people use them so they can get mail no matter where they are. Lots of these people will be fundamentally stuck under your proposal, because every time they used a kiosk they'd have to enter in all 20-30-40 POP servers they had mailing lists on. Surprisingly enough, hotmail.com and yahoo.com have a lot of subscribers, you might want to factor that into your plan pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
. It is quite a low bandwidth operation (probably less than 1K bytes) to poll a POP server for email It's not the bandwidth - it's the fact that there are these annoying things called timeouts. For *each* server that isn't reachable, you get to wait a minute or so - suddenly those 150 checks are taking quite some time. And the queuing theory of 150 sites with intermittent connectivity doing a push to your POP server is different than what you see when you try to do 150 pulls First, you are using a baseline of 150 in your example, which is attuned perhaps to your power use given you stated that you manage 6,000 mailing lists. I doubt the average user would subscribe to more than 10 on a regular basis. Second, there is a way of dealing with timeouts on the client side, it is called multithreading. Multithreading (for those readers who might not know, if any) is a fundamental programming concept that all internet clients (and all computer programs and operating systems) depend on. Thus the culmulative delay is not O(n), it is more like Max(n). Thirdly, with your queuing point, you are making the mistake of assuming that the majority of email received is *legitimate* bulk email. I assert this is far from the case for the majority of people. Any way, for the power cases like yours, what is the big deal with setting your email client to run for a few minutes each day, while you go browse some research at google.com or fill your coffee cup? Is your personal paradigm of immediacy for legitimate bulk email a good priority for the internet as a whole, when the very nature of email itself is being doomed by spam? But of course, if you actually *tried* this, you'd understand it... Please stop the personalized attacks and stick to the facts. the same messages over and over again. This would either require a special modification to the POP server and require each user to login with a unique user name. Hey.. what did I tell you? Everybody needs a login of their own... Er...you conveniently ignored the or that followed the either: -I wrote (and you ignored)- Or better, users' email clients can be made smarter because there is a UIDL command in both POP3 and IMAP4. This unique identifier can be used by the email client to only download messages which are new to that user. One would assume that POP servers could also remove messages older than say 1 month or so (configurable by the administrator). And as a side benefit, there would be no way for someone to subscribe me to a list without my permission, as can be done by sniffing an authentication email for Majordomo. If your confirmation mail is being sniffed, you have *BIGGER* issues. Again you conveniently ignored the overall point which followed. -I wrote (and you ignored)- And no way for someone to subscribe me to a list that has no public instructions for subscribing or unsubscribing (i.e. spam in guise of business email). That seems to be a BIG problem that many people have these days, probably even yourself. The flags can either be stored at the POP server and then give each user a unique login id, Hey.. there's that unique login again. Er...you conveniently ignored the or that followed the either: -I wrote (and you ignored)- , or more realistically just let email clients manage their own flags using UIDL. No only 6000 POP accounts. See above how email clients can handle the detection of new messages using UIDL. And you only need one anonymous login and no password (just configure the POP server to accept any login and password). And I tell *MY* UIDL from Keith Moore's UIDL from Vernon Schreyer's UIDL how? Yes your email client can keep track of which UIDL's it has already downloaded. Keith's email client can keep track of his. Vernon his. Ironically you mention Vernon (is that a hint?), because if I remember correctly he runs the DCC and recently had a similar unexplained overtly negative reaction recently when I tried to contact him regarding some improvements to DCC. I suppose it is possible that your negative responses here may be explained by similar vested interest with him. Maybe, but in fairness I won't immediately assume the worst ethics of you. BTW, in case you think my anti-spam business depends on this proposal, that would be false assumption. Actually one of the key advantages of my anti-spam algorithm (the automatic white listing of legitimate bulk email) would be rendered unnecessary if this proposal was put into effect. Have you actually *TRIED* to use more than 100 POP accounts under any current mail software? I will respond with similarly rhetorical question. Did you try to use Netscape 2 on most current web pages? Why make any application RFC if there can be no progress in applications? There's a difference - I'm not proposing a new scheme of doing things that
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Another reason why you need unique userid/logins for each subscriber - so that you can prevent forging a UIDL for somebody else to keep them from reading the message. Being able to do this (and if you have a shared userid, it's almost impossible to prevent) would make the Bernstein/Bush flamefest regarding list censorship look trivial in comparison... Again I will repeat: You apparently did not understand the concept. The per user UIDL tracking is stored in the email client, not in the POP server. Oh... there's also this thing called webmail. Lots of people use them so they can get mail no matter where they are. You can get mail no matter where you are with a POP account also. Most people use webmail either because it is free, or because it is a convenient interface on their POP account. Lots of these people will be fundamentally stuck under your proposal, because every time they used a kiosk they'd have to enter in all 20-30-40 POP servers they had mailing lists on. Again cars today have rubber tires and titanium rims, not wooden wheels. Applications can improve. It actually happens most of the time. Hotmail and Yahoo already support and encourage the pulling of email from POP accounts. You don't have to enter the POP accounts every time, just once. Surprisingly enough, hotmail.com and yahoo.com have a lot of subscribers, you might want to factor that into your plan How many more silly personal attacks?
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 11:51:19 +0800, Shelby Moore said: Hotmail and Yahoo already support and encourage the pulling of email from POP accounts. You don't have to enter the POP accounts every time, just once. Hmm.. so Hotmail is willing to maintain the list of 40 or 50 places it has to POP your mail from for you? Or are you re-entering it every time you use a new kiosk someplace? (Remember that the major business case for webmail is that you have a *VERY* small amount of state carried with you (a userid/password pair - specifically so you can check mail with a minimum of local resources). pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 11:38:19 +0800, Shelby Moore said: Please stop the personalized attacks and stick to the facts. I didn't make a personalized attack. I merely pointed out that in all probability, you haven't actually *tried* doing what you suggest, because it's not anywhere near as usable as you might think. ... and then... So you are indeed heavily involved with the DCC! Yes I know that is a weakness in the way the DCC measures bulk, but that is irrelevant to the point above. If I'm involved with the DCC in any way shape or form other than the fact I've seen Vernon write about it on IETF lists, it's such a covert involvment that even I am unaware of it. pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
--On Saturday, September 06, 2003 8:22 PM +0800 Shelby Moore [EMAIL PROTECTED] wrote: Request for opinions on whether to creating a working group or publish the following idea as an internet draft? Spam is big problem that is getting worse. BrightMail.com (which claims to process 10% of world's email) claims that the percentage of spam out of all email has grown from 16% in Jan. 2002 to 50% in Aug. 2003. A fundamental unsolved problem of doing any thing about spam, is there is currently no unambiguous definition of spam as an enforceable internet standard. This has been architectually impossible to define because the receiver is the subjective determinant of which bulk email is solicited and which is spam (UBE). ... Shelby, Valdis has identified some of the technical issues associated with using POP3 in this way. Let me step back and look at your proposal from another angle. From the standpoint of the bulk mailer/ poster of material, there is no advantage (and some disadvantages) of doing that posting so that interested users can pull it relative to just posting the material on a web page somewhere. Functionally, what you have proposed seems, to me, to be roughly equivalent to: * distributors of bulk materials are required to post them on selected web sites. * non-bulk materials may continue to go out via conventional email. Nice plan. The problem is that spammers won't play and efforts to coerce them into playing will largely fail due to international issues, lack of adequate incentives, etc exactly the same problem we have today with state laws prohibiting spam. More generally, you have just defined an opt in model that assumes that anyone who has not explicited opted to receive particular messages will be able to get them (or be sent them) only be some overt action on the would-be recipient's part. We know from experience that such a model won't work without significant legal pressure and enforcement -- if you don't believe me, sample any reasonable quantity of spam for messages that claim, quite strongly, that, if you hadn't opted in, you wouldn't be receiving it. Sorry, but no cigar. john
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
At 11:53 PM 9/6/2003 -0400, you wrote: Actually, the point is that there was no way, even within usenet, to prevent pollution of individual groups with innappropriate spam or off topic messages. Many groups have fallen into disuse for this reason. Afaics, that is irrelevant, because even a mailing list can be polluted with spam. Your point is against the nature of a public list or group, but my proposal is not designed to fix that problem. My proposal is designed to fix the problem of receivers being forced to receive bulk email from any sender. My proposal forces email addresses to not be public to legitimate bulk email. And then all other bulk senders can be dealt with more effectively. Then of course this does solve the problem for mailing lists also in the end, because then all bulk email gets killed (by a combination of legislative, judicial, ISP, Host, and software features). The fundamental problem with doing any thing about spam, is there is no way to measure it, because the subjective measurement of unsolicited is not architectually defined. The point is that if I choose to receive email from list or group (irregardless of whether that list or group has effective policies to prevent incoming spam posts), then it would be helpful to *architectually* differentiate that which I subjectively decide is legitimate bulk email senders from that which I decide is UBE (spam) senders. Once you define the architecture, then all kinds of important features can be built on top of it to deal more effectively with spam ranging from legislative, judicial, ISP, Host, and software. The other point to understand is that the paradigms (usenet vs email) are and have been different for very good reasons. And if one assumes that both paradigms are desired, then it is necessary to address how one enforces the separation. No one to date has figured out even a set of definitions to address the needs, much less mechanisms. I am unclear how you think this applies to my proposal? Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
I merely pointed out that in all probability, you haven't actually *tried* doing what you suggest, because it's not anywhere near as usable as you might think. How could you know if does not yet exist? We can discuss facts and theories of how it would work. But no one can actually test how it does work, because it does not exist yet. The comparison of setting POP accounts in an email client and popping the email, versus subscribing to that many bulk email lists and popping the email, is not useful because no email client has yet been designed to optimize popping from many anonymous POP accounts. If I'm involved with the DCC in any way shape or form other than the fact I've seen Vernon write about it on IETF lists, it's such a covert involvment that even I am unaware of it. Ok cool. Let's move on. Shelby Moore http://AntiViotic.com
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
Valdis has identified some of the technical issues associated with using POP3 in this way. I have refuted all of Valdis's technical points so far. Let me step back and look at your proposal from another angle. Yes I think that is productive to discuss the end game (outside of technical issues). From the standpoint of the bulk mailer/ poster of material, there is no advantage (and some disadvantages) of doing that posting so that interested users can pull it relative to just posting the material on a web page somewhere. Functionally, what you have proposed seems, to me, to be roughly equivalent to: * distributors of bulk materials are required to post them on selected web sites. It is an acceptable analogy, except I disagree that bulk posters of material favor web post over POP post. I think they favor what ever their receiver favors, by the law of supply and demand and economics. If receivers find it more convenient to get a copy in their InBox, then that will be accomodated (just as with mailing lists and archives now). But in terms of the analogy, I will follow along... * non-bulk materials may continue to go out via conventional email. Agreed. Nice plan. The problem is that spammers won't play Agreed. and efforts to coerce them into playing will largely fail due to international issues, lack of adequate incentives, etc exactly the same problem we have today with state laws prohibiting spam. Here is where I *strongly* disagree. Let me start with a story. The genesis for this proposal came from the fact that our outgoing business email (not bulk but single emails sending a password to a customer, etc) is being blacklisted by SPEWS (etc) because Rackspace (our Host) allowed some other customers of theirs to send spam on the same C class IP range as ours. SPEWS then blacklists the entire C class. Well SPEWS is uncorrectable lately because they've been under DoS attacks (from spammers presumably), thus caches of blacklists are used and nothing can be done except for us to change our email IP address. So in discussing this with the AUP manager and her boss at Rackspace, it became clear that Rackspace would never be able to guarantee the quality of a C class range of IPs, because Rackspace can not determine what is legitimate bulk email and what is spam and thus can not terminate new customers until a very heavy proof of spamming has already occured, by which time the damage to C class has already been done. So the lesson learned was that if Rackspace could automatically detect high quantities of bulk email in real-time, then with my proposed architectual change, Rackspace could in real-time shut off the spammer. Okay so that is one example of how the architectual paradigm changes the rules and allows more effective actions against spammers. Now take for example legislative combined with ISP. For right now, spammers are avoiding open relays and many foreign IPs because of blacklists, so they get a dialup account and send from there. Well if there was a law requiring USA ISPs to detect and shut these off in real-time, then spammers would need to revert back to open relays and foreign IPs which are effectively dealt with using blacklists. Then blacklists would not have to be so draconian with IP C ranges in countries with strict enforcement, which would make the blacklists more effective and useable. Then take anti-spam software like the DCC, BrightMail, or even our AntiViotic.com. If we know all bulk is bad, the game gets simpler because no whitelist needed. Since whitelists are data that is forgeable by spammer, this closes another hole. No to mention that whitelists make current anti-spam less useable and realistic on wide scale. I could go on... but I hope you begin to see how everything to fight spam depends circularly on the ability to architectually define it. If you can't measure it, you can't do any thing about it. That is a fundamental datum of science. More generally, you have just defined an opt in model that assumes that anyone who has not explicited opted to receive particular messages will be able to get them (or be sent them) only be some overt action on the would-be recipient's part. That is the definition of opt-in. We know from experience that such a model won't work without significant legal pressure and enforcement -- if you don't believe me, sample any reasonable quantity of spam for messages that claim, quite strongly, that, if you hadn't opted in, you wouldn't be receiving it. What you are saying IMO, is that you can't force bulk emailers or spammers to use opt-in. That has been because you can't measure the spam (UBE) from the legitimate. It is a chicken and egg problem. Once you have the egg (the architectural metric), then reasonable to make the chicken. So comparing to before you had the egg, is not necessarily illustrative. Sorry, but no
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 12:23:19 +0800, Shelby Moore said: Interestingly note that Hotmail makes you pay to POP *FROM* hotmail, but no charge to POP from other accounts *TO* Hotmail. Does that give you any hint about their business model?? Yes. It's *NOT* a business model where they want to be polling a dozen servers on a regular basis for each of their customers for mail that may or may not be there, and for the average mailing list, probably is not there at any given poll. They want eyeballs, and the last thing they want to do is expend more effort than needed to get eyeballs. Sure - they can even optimize the 'POP the list' check by only doing it once for all the subscribers - but they're still hitting each server for each list on a several-times a day basis. And under the current scheme, they can just *catch* one SMTP transaction with all the RCPT TO's piggybacked *when there's actual mail*. So they'd have to work a lot harder under your scheme. And let's *THINK* for a moment here - what is your proposal *REALLY* going to change? We already have many estimates that 50% or so of all e-mail is spam. Let's take that as a given, and let's make the rash assumption that the rest is 25% mailing list traffic and 25% person-to-person. So what you want to do is take the 25% of the list traffic that works just fine on the current infrastructure, and is usually quite easily whitelistable via a number of different methods - and move it to something totally different. And what you're left with is a 2-1 mix of spam and personal mail that you yourself admit things like the DCC and spam filters are unable to perfectly distinguish. To quote Douglas Adams: Many reasons for this unhappiness were proposed, most of which involved the movement of small green pieces of paper, which was odd given that on the whole, it wasn't the small green pieces of paper that were unhappy... Having exiled the mailing list traffic, we would then be able to work on separating the spam from non-spam - but as you already noted yourself, we don't know how to do that yet. And getting rid of the mailing list traffic doesn't in fact gain us anything at all, since everybody who filters list traffic into separate folders for each list knows that isn't the problem - it's the unfiltered stuff that's left in the inbox. I'll note in passing that the two highest SpamAssassin scores I've ever seen were both on legitimate postings to mailing lists - both were humor pieces about spam Quite frankly, given that at least half the spam I get is already in obvious violation of at least one law (pick one - securities fraud, advance-fee scams, wire fraud, bogus pharmeceuticals, or hijacking a proxy to send the mail), I severely doubt that anything the IETF does in regards to standards won't make a difference. The spammers often don't even bother following RFC822 - why should they follow your scheme? The *only* two ways to get rid of spam both involve making it non-profitable. The first is lowering the generated income. Given that recently, somebody hacked the site of a fertilizer for your body part scam, and found a list of 6,000 people who had paid $50 a bottle, I have to sadly conclude that Korbluth and Barnum were both correct, there's one born every minute and the rate is increasing. So there's no joy to be found there. The second is raising the cost to the spammer. Personally, I like the idea of taking up a collection among the ISPs and other providers, and hiring some good ethnic muscle (there's competition in the field, a number of experienced and ruthless groups are available). I'm sure the spam problem would change drastically if the spammer was seriously having to balance the mentioned $300K for bogus enhancement pills against having their kneecaps broken by one group or worse by one of the other groups... Pity that will never work though. At least not officially (although one infamous New Zealander apparently retired recently...) pgp0.pgp Description: PGP signature
Re: Proposal to define a simple architecture to differentiate legitimate bulk email from Spam (UBE)
On Sun, 07 Sep 2003 13:07:10 +0800, Shelby Moore said: It is a wrong assumption to equate commercial email with bulk email. Which is why you're trying to rewrite how bulk email is done in order to deal with *one segment* of commercial e-mail. Now I understand fully. pgp0.pgp Description: PGP signature
