Re: covert channel and noise -- was Re: proposal ...
Dean Anderson wrote: A covert or sneaky channel is merely one in which the communication is //not authorized by the security model// It has nothing to do with readability or detectability. To be useful for its covert purposes, a covert channel should not be easily detectable as a covert channel -- in our case, spam should not be easily detectable as spam. Thus, it has to do with readability and detectability. That's why, as a matter of logic, we should agree that if the message can be detected/read by the intended recipient then it's not in a covert channel (anymore). This does not imply that if the message can NOT be detected/read by the intended recipient then there is NO covert channel. In yet other words: you whack-a-mole when you find one, but you can't say that there aren't any moles. It is not a game you can win. That's where we disagree -- if I can make it harder/slower for the other side to set up moles than it is for me to find them, I will win. We now have a legal process to use against abusers someone must have said the same for anything we have a law for... including theft...and yet we do find it useful to lock our cars, no? In many residence areas, mailboxes have no key and anyone can open them and insert rogue mail, bombs, etc. This is the same way that your email address works today. Anyone can put email in your emailbox and, if they're clever enough, spam filtering in your personal MUA or at the MTA will not help. You will receive spam. You say this is not a game you can win... there is little that can be done. In some residence areas, however, mailboxes have no mail slot. The mailman has a key to the mailbox and needs to use it in order to insert mail. The box owner has another key and uses it to retrieve mail. This still does not prevent you from receiving a letter laced with anthrax, but it gives us a good metaphor for improving email: We need to be able to control receiving what is posted to us based on the trust/authorization we associate with the poster AND the message. In other words, if we have no reason to trust the poster or the message, we should be able to impose a burden as high as we desire on the poster (including no burden). The decision what to accept or reject should happen at the recipient's MTA (preferably) or MUA in interaction with the purported sender's MTA and MUA. Also, to be effective, this should not depend on a user's list of current va-like words. Providing a barrier for accepting email (i.e., a putting a selective lock in your email box) is neither a legal issue nor an user issue -- is an IETF issue. By using suitable PK encryption for end-to-end email privacy (including crypto-puzzles), I suggested we can offer such spam burden at no added cost to the user. I advance that in the old DARPANET days, spam protection was provided by DARPA, because DARPA could locate and disconnect any user who abused the system, and everyone knew it. That's why there was no need to design email in any other way than what is today, with open mail addresses. However, today, the Internet is an open system and there is no way to locate and effectively disconnect abusers. The abusers will just continue to route around, seeking zero friction to posting, until we put in place a better system just like the postal mail had to do, laws notwithstanding. Cheers, Ed Gerck
Re: covert channel and noise -- was Re: proposal ...
Vernon Schryver [EMAIL PROTECTED] wrote: I know of many millions of spam that are filtred during the DATA command every day, and I don't claim to know about any really big sites. The only problems are: - local administrative choices that keep bastion SMTP servers ignorant of per-user filter preferences This is a feature, not a problem. If the end user wants a filtering process individualized that much, s/he should choose to use a SMTP server which agrees to do so. - filtering at the DATA command requires either (1) rejecting for all or no targets or (2) accepting for all targets and siliently discarding the message for those targets that want it filtered. Alternatively, the receiving SMTP server could reject any multiply- addressed email. Is it actually that unreasonable to apply the most-restrictive filtering rules in the case of multiply-addressed email? (Silently discarding _is_ a bad idea, when done by the SMTP server itself. IMHO, it's better to mark for later discard -- which actually could be done in such a way as to mark only for those recipients who requested the more restrictive filtering.) In theory the second problem could be fixed if the DATA command could accept a vector of 250-OK/4yz-try-later/5yz-fatal responses, one for each target named with a Rcpt_To command. In practice the spam problem will be solved one way or another long before such a protocol change would be sufficiently widely deployed to matter. Agreed: that radical a change in SMTP wouldn't percolate through quickly enough. -- John Leslie [EMAIL PROTECTED]
Re: covert channel and noise -- was Re: proposal ...
From: John Leslie ... - local administrative choices that keep bastion SMTP servers ignorant of per-user filter preferences This is a feature, not a problem. If the end user wants a filtering process individualized that much, s/he should choose to use a SMTP server which agrees to do so. That is a feature only if the user accepts the consequences of discarding mail without generating bounces, including not informing senders of false positives. Bounces from internal spam filters (either in MUAs or MTAs inside organizations) are a major source of unsolicited bulk mail or spam. - filtering at the DATA command requires either (1) rejecting for all or no targets or (2) accepting for all targets and siliently discarding the message for those targets that want it filtered. Alternatively, the receiving SMTP server could reject any multiply- addressed email. People running SMTP servers that handle 100K or more msgs/day have been uniformly horrified when I've suggested that. I don't really understand why, but I have given up on the idea. (Silently discarding _is_ a bad idea, when done by the SMTP server itself. IMHO, it's better to mark for later discard -- which actually could be done in such a way as to mark only for those recipients who requested the more restrictive filtering.) A better positition is that everything should be logged, particularly including discarded mail, and in that case, enough of bodies to allow targets to identify senders and the nature of the discarded messages. Of course, one should assume users won't normally look at those logs. Spam you read is not filtered, but at most categorized and stigmatized. Vernon Schryver[EMAIL PROTECTED]
Re: covert channel and noise -- was Re: proposal ...
From: Robert G. Brown ... Or, mark for later accept/reject decisioning AFTER the SMTP server per se, in the filter pipeline between the server and the mail spool of the addressee. Spam assassin does the right thing already (and this is exactly what it does). ***NO***! Except when run as a milter or otherwise during the SMTP transaction, SpamAssassin does the WRONG thing. As run almost everywhere, after the SMTP transaction, SpamAssassin can only iether silently discard spam or generate new spam by sending bounces to innocent people. A better positition is that everything should be logged, particularly including discarded mail, ... Logging a message you reject is nearly a waste of time. Based on my experience, people running ISPs or other large mail system strongly disagree with your position. Besides, I intentionally wrote about logging ***discarded** mail. Many institutions do turn off logging of greylisted messages, reduce the default per-message logging limit of 32K Bytes, or delete log files far sooner than the 14 default in the DCC source. Still, logging is seen as vital to be able to answer questions about which messages were filtered and why, including being able to say that message was never sent or substantially identical copies of that message were sent to 310 other users here and 433,797 users elsewhere; it was spam. In order to recover the message (as you note, nobody ever looks at the logs, which are VERY LARGE for a busy mailer and beyond human capacity to scan), I said nothing about humans scanning everything. Besides, giving users the sense that they can see what's happening with spam filtering on their mail and control it is a requirement for getting users to accept filtering. .. This is where, and why, I take issue with filtering and discarding at the level of the SMTP server, unless the accept/reject decision can be made with 100% precision (no false positives, no false negatives, and it may not be good even then because MY idea of the correct basis for the decision may not be the same as YOURS). What you describe is a broken version of what I advocate, if you consistently look at your personal log of rejected mail. Your version is broken because a reject decision after the SMTP transaction must at least sometimes result in sending spam to innocent people. ... It's not that filtering based on non-header-linked aspects of content is or isn't a good idea in some cases. It is that it has no business being in the specification of TCP. ... pure chance have a byte sequence like SEX that caused it to be rejected .. Nice straw man. I've never heard anyone with a taint of technical clues talk about looking for SEX in raw TCP segments. ... For nearly all filtering programs, it is too easy to create a message that is filtered but shouldn't be. ... You evidently lack experience with the filters used by commercial institutions. Commersical SMPT servers cannot tolerate false positives (legitimate rejected/total legitimate) of more than 0.01%, and even that's pushing it. The design requirement for filtering mail on which money depends is that false positives must not be much worse than the underlying SMTP error rate due to problems such as full disks and broken DNS servers--not to mention mail recipients too quick to delete. A lot of current talk about false positives is self-serving nonsense from such as the Direct Marketing Association. Manual spam filtering also has false positives. A human suffering a common spam load of 100 spam/day has trouble not deleting legitimate mail. My filters are rejecting about 300 spam/day sent in my direction, 12227 in the last 40 days. Mechanical filtering even with a significant false positive rate can reduce the overall false positive rate. ... SMTP was designed to permit reasonably RELIABLE (simple) transport of It would be good to skip the networking 101 tutorials. Those of us who don't know all of that about TCP, SMTP, CSMA/CD, etc. often thanks to decades of personal experience should apply elsewhere to learn the basics. It is may hard to imagine how old farts like me see such tutorials, but please try. I've been receiving email as vjs since 1968. It seems to me to be highly unacceptable to attempt to insert content-based accept/reject decisioning in at this PROTOCOL level in the delivery process. That use of level confuses me. It does not seem to conform to ISO OSI architecture. ... reliable transport mechanism for important messages. Filtering it for me according to ANY CONTENT-BASED RULESET risks discarding at least some messages that are not correctly classified when they are rejected. Important messages can be lost. Bad things can result. Who is responsible when this occurs? Who do I get to sue? You don't get too sue anyone, because a reasonably designed system lets you choose to do all of your spam
Re: covert channel and noise -- was Re: proposal ...
Robert G. Brown [EMAIL PROTECTED] wrote: THIS IS NOT A MATTER OF PROTOCOL! This is not the business of the IETF! Tools exist to help you filter spam. Some, like spam assassin, are very good and quite sophisticated. However, there is ALWAYS a tradeoff with using this sort of tool -- too narrow a criterion for acceptance and you lose real mail, too broad and you get all the spam anyway. I can choose my level of tradeoff, and be fully aware of the risks. While the parameters of individual spam-filtering _should_ not be the business of IETF, there is a very real issue raised by filtering _after_ the SMTP session: During the SMTP session, it is quite possible to give an error which is likely to reach the right person; after the SMTP session, you have to trust the various header fields -- which are routinely forged by spammers. In fact, we are now facing a _second_ denial-of-service problem (the first being spam itself clogging your mailbox): excessive bounce messages automatically directed to mailboxes forged by spammers. And this by its nature is a distributed-denial-of-service problem, even harder to protect against. Spam-filtering _after_ the SMTP session _should_ be deprecated by IETF, IMHO. While there is no question that any recipient has the right to ignore any message, SMTP was intended to either deliver the message or send back an error; and the loss of this feature reduces the utility of e-mail. -- John Leslie [EMAIL PROTECTED]
Re: covert channel and noise -- was Re: proposal ...
In fact, we are now facing a _second_ denial-of-service problem (the first being spam itself clogging your mailbox): excessive bounce messages automatically directed to mailboxes forged by spammers. And this by its nature is a distributed-denial-of-service problem, even harder to protect against. I agree, and in fact pointed out that bounce messages from AV programs are similar problem a week ago. So perhaps the discussion should separate into three different threads: a) Encryption and host authentication on the smtp channel. This might embrace open mail programs used as forwarding agents by both spammers and viruses. b) What to do about bounces in general. Bounce messages are useful or even essential in many contexts, but they are a source of both intentional and unintentional abuse in a world where headers are routinely forged. c) Spam, what (if anything) to do about it at the IETF level. It may be that good solutions to a) and b) may, together with some support in the form of laws, suffice to greatly ameliorate both spam and header-forged viruses by increasing the accountability of the enabling networks. Most of this traffic is ALREADY in violation of corporate acceptable use agreements -- part of the problem is that many ISP's have been very lax in enforcing acceptable use and tracking down violators. Part of THAT problem is that the networks selling access to the ISP's have in turn turned a blind eye to the ISP's. Perhaps a header-level/protocol-level solution would be a generalization of the postmaster address that points to a HUMAN responsible for policing the network(s) where spam/viruses originate. Yes, one could argue that postmaster already is such an address, but many of the smaller originating networks are run by amateurs and have no real postmaster. One really needs to be able to hit a recursive list of postmasters along the message's delivery route with warnings about violations of acceptable use. Spam-filtering _after_ the SMTP session _should_ be deprecated by IETF, IMHO. While there is no question that any recipient has the right to ignore any message, SMTP was intended to either deliver the message or send back an error; and the loss of this feature reduces the utility of e-mail. I agree. However, all of the observations made so far regarding spam/virus filtering in general still apply to filtering at the SMTP level. I would say that NO keyword filtering could take place. The best that one could do at the protocol level would be to reject messages that fail to meet a tighter criterion than is currently required. Perhaps: a) All hosts must resolve with DNS. b) All hosts must support an encryption key registered with DNS that permits all message hops to occur between registered hosts encrypted with the destination host public key. c) The header autogenerate a postmaster-recursive email address for reporting abuse to the entire delivery path. This would put a rather large burden on the main network backbone administrators -- they'd need automated tools to help handle it. OTOH, it would REALLY give them an incentive to shut down networks that are a primary source of abuse until they manage to police themselves. Automated tools at the highest level might just generate an abuse histogram that triggers a human response only when levels rise above that which might be identified as reasonable for a network participating in the eternal battle with the bad guys. d) With keyed host registration, tools that can QUICKLY isolate an originating host and bop its (ab)user (minimally get them off the network, ideally instantly fine them or charge them money such as a reconnection fee AFTER getting them off the network). This would give end users a strong incentive to police their own systems against viruses and would give spammers additional costs to pay or additional charges to be brought against them, should they try to skip out. I personally would ALSO like it if AV vendors STOPPED bounce messages altogether. SMTP bounce messages have a point and are even necessary; AV bounce messages are a joke, a waste of time, and a potential source of serious abuse. NO viruses currently exist that don't forge the header, it is just that most viruses nowadays use random forged headers out of the infected host's address books. They could, however, all claim to be from e.g. SCO or Microsoft (some already exist that do the latter, but more for social engineering reasons than DDOS, I think). rgb -- John Leslie [EMAIL PROTECTED] Robert G. Brownhttp://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:[EMAIL PROTECTED]
Re: covert channel and noise -- was Re: proposal ...
From: Robert G. Brown [EMAIL PROTECTED] ... I agree. However, all of the observations made so far regarding spam/virus filtering in general still apply to filtering at the SMTP level. I would say that NO keyword filtering could take place. The best that one could do at the protocol level would be to reject messages that fail to meet a tighter criterion than is currently required. What is the SMTP level? Is that during SMTP transactions between MTAs? Is the protocol level the same or something else? If both of those levels refer to SMTP transactions between MTAs, then the conclusion is wrong. Except for local administrative hassles, all spam and virus filtering that can be done later can instead be done during SMTP transactions between MTAs. Your SMTP server may need to wait to until the end of the DATA command to object to messages containing viruses, missing or bad SMTP headers or other objectionable content, but that works fine. I know of many millions of spam that are filtred during the DATA command every day, and I don't claim to know about any really big sites. The only problems are: - local administrative choices that keep bastion SMTP servers ignorant of per-user filter preferences - filtering at the DATA command requires either (1) rejecting for all or no targets or (2) accepting for all targets and siliently discarding the message for those targets that want it filtered. In theory the second problem could be fixed if the DATA command could accept a vector of 250-OK/4yz-try-later/5yz-fatal responses, one for each target named with a Rcpt_To command. In practice the spam problem will be solved one way or another long before such a protocol change would be sufficiently widely deployed to matter. Vernon Schryver[EMAIL PROTECTED]
Re: covert channel and noise -- was Re: proposal ...
Robert G. Brown wrote: On Sun, 15 Feb 2004, Ed Gerck wrote: We can't lock the spammers' doors everywhere, we have to lock our door at our house. No, what we can do is the same thing we do with our real mail box. Make it illegal to send certain classes of mail, for example letter bombs and envelopes containing anthrax, and prosecute the hell out of anybody we catch who does so. On the flip side, this is a good example of how the anthrax threat was not handled. The solution was not to rely on the illegality of sending anthrax (which, obviously, wasn't enough of a deterrent) or the probability of prosecuting the sender (which we have not yet done), but on a lock that was put on all mail coming in. Mail that came from unknown senders was more carefully verified and even sterilized, some was backtraced and the sender was verified. But postal mail, contrary to email, cannot put a burden on the sender that is higher than the postage cost. Email, rather than being cost-free for all senders, can be made expensive to senders we don't know. And the way to do it mandatorily is by using code -- not law. This is useful because our email is laced with the anthrax called spam. Filtering by subject line, email headers and body text is not enough, and frequently backfires by making us delay and even not read what would otherwise be a message of interest. BTW, the technique of imposing a task that burdens the sender in order to reduce interference due to rogue senders has a parallel in spread-spectrum technology, for example. By only receiving messages that are frequency-keyed as expected, my receiver can withstand jamming (i.e., noise messages that are in-band -- spam). This is in addition to other filters I have for post-processing.
Re: covert channel and noise -- was Re: proposal ...
Robert G. Brown wrote: a) All hosts must resolve with DNS. If you list why this isn't used today perhaps you will change must to may. b) All hosts must support an encryption key registered with DNS that permits all message hops to occur between registered hosts encrypted with the destination host public key. Mail privacy can only be guaranteed with an end-to-end encryption. Securing email in message hops does nothing to prevent monitoring at each host in the hop -- with some hosts not even advertised in the header. c) The header autogenerate a postmaster-recursive email address for reporting abuse to the entire delivery path. This would put a rather large burden on the main network backbone administrators -- they'd need automated tools to help handle it. OTOH, it would REALLY give them an incentive to shut down networks that are a primary source of abuse until they manage to police themselves. This would create a huge liability for the backbone administrators -- for example, one false abuse report and they could be sued for disrupting lawful communications. Human supervision actually increases the liability -- it can't be blamed on a software glitch. d) With keyed host registration, tools that can QUICKLY isolate an originating host and bop its (ab)user (minimally get them off the network, ideally instantly fine them or charge them money such as a reconnection fee AFTER getting them off the network). Machines running amok, quickly killing off other machines without recourse, without explanation. A kangaroo court for email, penalizing the users. This would give end users a strong incentive to police their own systems against viruses and would give spammers additional costs to pay or additional charges to be brought against them, should they try to skip out. Again, what you propose is to penalize the victim -- the user. That's exactly what we should stop doing. I personally would ALSO like it if AV vendors STOPPED bounce messages altogether. Free speech, good luck.
Re: covert channel and noise -- was Re: proposal ...
Robert G. Brown wrote: a) All hosts must resolve with DNS. If you list why this isn't used today perhaps you will change must to may. b) All hosts must support an encryption key registered with DNS that permits all message hops to occur between registered hosts encrypted with the destination host public key. Mail privacy can only be guaranteed with an end-to-end encryption. Securing email in message hops does nothing to prevent monitoring at each host in the hop -- with some hosts not even advertised in the header. c) The header autogenerate a postmaster-recursive email address for reporting abuse to the entire delivery path. This would put a rather large burden on the main network backbone administrators -- they'd need automated tools to help handle it. OTOH, it would REALLY give them an incentive to shut down networks that are a primary source of abuse until they manage to police themselves. This would create a huge liability for the backbone administrators -- for example, one false abuse report and they could be sued for disrupting lawful communications. Human supervision actually increases the liability -- it can't be blamed on a software glitch. d) With keyed host registration, tools that can QUICKLY isolate an originating host and bop its (ab)user (minimally get them off the network, ideally instantly fine them or charge them money such as a reconnection fee AFTER getting them off the network). Machines running amok, quickly killing off other machines without recourse, without explanation. A kangaroo court for email, penalizing the users. This would give end users a strong incentive to police their own systems against viruses and would give spammers additional costs to pay or additional charges to be brought against them, should they try to skip out. Again, what you propose is to penalize the victim -- the user. That's exactly what we should stop doing. I personally would ALSO like it if AV vendors STOPPED bounce messages altogether. Free speech, good luck.
Re: covert channel and noise -- was Re: proposal ...
On Sun, 15 Feb 2004, Ed Gerck wrote: Dean Anderson wrote: It isn't the case that the spammer intended to send a message about the superbowl, but somehow noise altered the message to a solicitation on viagra. Rather, they intended to send a message on viagra, and you recieved their message, noise free. But seeing the solication for viagra, you became upset, and reported a complaint about the inappropriate use of the channel. In information-theory-speak, you report a communication in violation of the security model; a covert or sneaky channel. I guess we agree that if the message can be read by the intended recipient then it's not in a covert channel. A covert channel is one that can't even be detected by the intended recipient. But, you may ask, who is the intended recipient? Err, we do not agree on that. You still misunderstand the nature of a covert channel. I suggest you re-read the definitions and references in the reposted messages from Ellis Cohen and Stavros Macrackis. A covert or sneaky channel is merely one in which the communication is //not authorized by the security model// It has nothing to do with readability or detectability. There is no theorem that says covert channels cannot be detected. Quite obviously, they are detected, and some action might be taken. Theory just says that you cannot prove they aren't there. Let me put it another way: A null reading on your covert channel detector does not mean there aren't any covert channels--Just that you haven't detected any. This is a subtle, yet significant difference. In yet other words: you whack-a-mole when you find one, but you can't say that there aren't any moles. It is not a game you can win. You have to keep looking and keep whacking, so long as there are moles to pop up. The arcade whack-a-mole game stops when you run out of time, but our game only stops when no one wants to spam or conduct abuse or government intervention prevents them from playing. We now have a legal process to use against abusers who are not commercial--most, if not all, genuine commercial spammers will simply comply with the law. That leaves those who aren't genuinely commercial, and whose intent it is to annoy people. An example of such a covert channel is if a spammer hides information in the subject line by using wrongly spelled forms of viagra, This could be a covert channel. But its also possible that the whole spam message even with a correctly spelled viagra would be a covert or sneaky channel if it is not //authorized by the security model//, and the security model is simply an Acceptable Use Policy statement not to do that. In this case, you may have a very weak covert channel detection process. But extreme weakness in the process is irrelvent to the theory. What, if anything, has to be done to get past the detection process and the security model and into your mailbox is irrelevant. The security model can be made quite extensive, such as with Mandatory Access Controls as implemented in secure operating systems. Even so, one cannot say that there aren't covert channels. One can say other things about them, but not that thing. Information theory has proved itself useful to the analysis of anti-spam schemes. Besides being able to rule out a number of schemes which promise to be complete technical solutions to spam, we also see what range of solutions can be implemented about spam and what results we can expect to see from that range. Consider the case of Mandatory Access Controls (MAC) for operating systems. We see that if we tighten up the controls on the flow of information, that we improve our chances of detection. Particularly, we hope to detect people trying to find covert channels before they succeed in finding one. This is very expensive, both in supervision costs, and in the difficulty of training workers to use such systems. In the case of classified government information, the cost is justified. Having a potential spy create a denied MAC security event while probing for a covert channel is well worth the effort, even if you can't be sure that they will be caught. The spy, or even potential spy or stupid user, is then removed from the secured areas. Even honest, but stupid users are removed, and this can be rationalized because they don't have the capacity to be trusted with sensitive information. It works because the same 'mole' doesn't get repeated chances to find the channel that will be successful, and there are legal processes to make that happen. You cannot lie on a security clearance application when it asks your identity, and if you've ever had a clearance revoked or denied. Expensive checks are made to ensure that your answers are truthful. But the same can't be said about spammers/abusers. We can't escort them out, and we can't even be sure of their identity in a civil context. Frequently, they are using someone else's identity or computer. We already detect them quite
Re: covert channel and noise -- was Re: proposal ...
On Sat, 14 Feb 2004, Ed Gerck wrote: Dean Anderson wrote: You are confusing a covert channel with noise. They aren't the same. Of course they aren't -- how could a channel be equal to noise, anyway? What I said is that the *information* transferred by a covert channel, whatever that information might be, is NOT the message (the intended information). So, the spammer didn't intend to send you a message about Viagra?? That somehow noise caused the message to be altered to be about Viagra?? I don't think so. Thus, for modeling purposes in terms of Shannon's information theory, the choice is clear. The *information* transferred by a covert channel is noise (the only other possible option). Not only isn't it 'the only possible option', but the information carried by a covert channel is not noise. The sender sent information and the receiver recieved the information. It isn't the case that the spammer intended to send a message about the superbowl, but somehow noise altered the message to a solicitation on viagra. Rather, they intended to send a message on viagra, and you recieved their message, noise free. But seeing the solication for viagra, you became upset, and reported a complaint about the inappropriate use of the channel. In information-theory-speak, you report a communication in violation of the security model; a covert or sneaky channel. A note of caution: The intent of the senders and receivers is irrelevant to information theory channels. Covert channels can exist with or without intent, and with or without cooperation. Information theory is concerned with the movement of information through a transmitter to a reciever. The intent or cooperation in that transfer is irrelevent, except that a channel used for information not intended for transmission is called covert or sneaky. For example, when a remote SSL attacker uses timing information to obtain the secret key of the server, the server isn't cooperating, nor does it intend to provide the secret key. Information theory looks at the actual value of the key, and whether that was transmitted to the reciever with or without noise. In the case of the SSL timing attack, there is a lot of noise, but information theory tells us, and practice confirms, that the noise can be corrected, until the key is successfully transmitted to the receiver error free. In the case of spam though, we safely assume the intent of sender to get a spam message to the recipient, and can couch our descriptions accordingly, but we need to exercise some caution. With regard to noise, what we are really be concerned with is the message before it is transmitted, and the message after it is received. In our case, the transmission is usually noise free thanks to the function of TCP. Antispammers have commonly used an analogy that equates spam with noise and anti-spam efforts as trying to find a noise filter. The analogy sounds good, but is not accurate, which might suggest a reason why they have failed to find a filter. If you are proposing a different model, fine. I'm not proposing a different model. The model of spam is noise that you are using is inaccurate and wrong. I'm explaining the correct way to model spam in information theory. This will help explain why many anti-spam proposals are unworkable, and why previous attempts have failed. Spam isn't unwanted until after the fact: You read it, and then you don't like it. I strongly disagree -- I don't read spam and I don't even try to read the unsubscribe information in it. I try the best I can to detect it and delete it as early as possible. If possible, even before it is queued in my mail box and causes me further problems (and costs). You don't _want_ to read it. You don't _want_ to spend time reading it. It's the fact that one doesn't have time to read all the junk that many people find annoying about spam, as well as objectionable content. But those who don't have email aren't annoyed by spam. You have to receive spam to be annoyed by it and not want it. I believe that's also what the majority of email users would like to do -- or, do you really think we all have the time to read spam and decide after the fact that it is indeed spam? What users want is constrained by what's possible. I want faster-than-light travel for free. Many people want that, which explains its popularity in science fiction. But we are constrained by physics and technology. Quite obviously, in the case of spam we are trying to find ways so that we don't have to read/sort/delete spam, or spend less time doing so. You have promised that your scheme will eliminate the whack-a-mole nature of spam. I'm saying that it is impossible to eliminate this charactistic from spam, based on what we know from information theory about covert or sneaky channels. This tells us that your scheme cannot deliver what you have promised. The prevalence of anti-spam schemes that claim to make spam
Re: covert channel and noise -- was Re: proposal ...
On Sun, 15 Feb 2004, Dean Anderson wrote: On Sat, 14 Feb 2004, Ed Gerck wrote: Dean Anderson wrote: You are confusing a covert channel with noise. They aren't the same. Of course they aren't -- how could a channel be equal to noise, anyway? What I said is that the *information* transferred by a covert channel, whatever that information might be, is NOT the message (the intended information). So, the spammer didn't intend to send you a message about Viagra?? That somehow noise caused the message to be altered to be about Viagra?? I don't think so. Thus, for modeling purposes in terms of Shannon's information theory, the choice is clear. The *information* transferred by a covert channel is noise (the only other possible option). Not only isn't it 'the only possible option', but the information carried by a covert channel is not noise. The sender sent information and the Once again, what Dean is saying is dead on correct. Information theory deals with corruption of bits, with the degradation of ordered information. In physics, entropy is the log of the missing information, where information has a very precise meaning, and with this definition much of statistical mechanics can be more or less derived from Shannon's theorem and a proper analysis of conditional probabilities. An alternative view of SPAM where it isn't a covert channel (which sounds a bit too much like spy thriller stuff, although I understand perfectly well what is meant) but rather considers it merely to be an undesireable communication clarifies things a great deal. What is undesirable is clearly a matter of opinion and requires the solution of a serious (largely unsolved) problem in predictive modeling or artificial intelligence in order to mechanically filter. I'd like to make a suggestion. There are several parts of this discussion that are quite distinct, and by mixing them (often out of context) the discussion is being perpetuated well beyond sane boundaries. First of all, there is the issue of whether or not email should routinely and USER TRANSPARENTLY be encrypted, at least during the transit phase between MTA's. This really deserves an entirely separate discussion, as I think one might well build up a consensus that ALL network traffic should be routinely encrypted at this point and that email in general is a glaring legacy exception in a world where SSL and ssh have largely dealt with encrypting the rest, at least where it most matters. This, in my opinion, is very much an IETF issue as it would require the development of an open protocol and the supported engineering of retrofitting MTA's everywhere to use it. This in turn may well hinge on broader issues of host registration and authentication. After all, most users of web browsers and ssh are not deeply aware of the key manipulations that take place in order to secure those channels EXCEPT when they are being attacked and threaten to fail. Surely something could be done with mail as well. Second, there is the issue of whether adding a cost to email would deter spam. I note that no one (Ed in particular) has yet attempted to rebut by assertion that doing the actual arithmetic it is clear that individuals emailing SPAM in excess of the legal limits triggered by e.g. the new Virginia anti-SPAM law (10K pieces a day, 100K per month) could care less if the COMPUTATIONAL cost per message were 8 whole seconds per mailing, so delays up to that level are irrelevant. 8 seconds is some 20 to 30 billion instructions on a modern CPU, and nothing restricts spammers to using just one. I think that it is absolutely clear that adding an encryption step as a cost to deter spammers is simply ridiculous as it will do no such thing. Adding a cost to an email message to all users (instituting per-message costs onto the internet backbone) is a cure so much worse than the disease that I, for one, will vehemently oppose it, as will nearly all old Internet hands. That way madness lies. And in any event, per message costs will simply shift the cost plateau around and make sending spam legitimate business with more than a handful of individuals making a profit from it. The moment spam is viewed the way TV advertisements are, as a perfectly reasonable adjunct to a medium, we are all doomed. I am by no means convinced that spam control is in any way the business of the IETF. If it is, it must be at the detailed engineering level, and the discussion is very, very far away from that so far (where people are routinely making statements like implementing this will result in that without any sort of quantitative analysis or basis. Engineering is about numbers, not just ideas. At the very least, ideas supported by actual numbers. In fact, before making ANY sort of proposal concerning spam, it would behoove the primary participants to study and use themselves on an active basis a wide range of the existing anti-spam tools. Second, in at
Re: covert channel and noise -- was Re: proposal ...
Dean Anderson wrote: It isn't the case that the spammer intended to send a message about the superbowl, but somehow noise altered the message to a solicitation on viagra. Rather, they intended to send a message on viagra, and you recieved their message, noise free. But seeing the solication for viagra, you became upset, and reported a complaint about the inappropriate use of the channel. In information-theory-speak, you report a communication in violation of the security model; a covert or sneaky channel. I guess we agree that if the message can be read by the intended recipient then it's not in a covert channel. A covert channel is one that can't even be detected by the intended recipient. But, you may ask, who is the intended recipient? In anti-spam email systems, we can usually recognize three whos: #1: My MTA #2: My MUA #3: Myself The spammer's strategy is to send the spam message in such a way that it is undetectable by my MTA-MUA, while it is detectable and readable by me. In short, it needs to use a covert channel through my MTA-MUA, but not to me. An example of such a covert channel is if a spammer hides information in the subject line by using wrongly spelled forms of viagra, information which my MTA-MUA cannot detect. It's not a covert channel for me but it is for my defenses. But, once that message passes through to me, it becomes detectable, readable and I call it spam. Does this mean that spam is defined by the rule I (only) Know It When I See It! and we have to accept a large failure rate in preventing spam, that can only be solved by laws and law enforcement? I want to emphasize that it does not have to be. Suppose a user can make email senders pay a burden at their MTA or even at their MUA (e.g., a bounce requesting encryption and solution to a puzzle). In addition, using a selective scale defined by the user (so that selected mail senders have less burden) at their MTA and MUA, the user can make the spammer pay a price *as high as desired* by the user (not limited by R. Brown's comments). In such case, the covert channel cannot even be established unless the spammer pays a price -- a price that can be *as high as desired* by the *user*. This is the essence of the proposal (the devil is in details). I take the example of the front door of your house. If you leave it open, so that a thief has no burden getting in, a thief probably will steal something from you -- even though the law says that theft is illegal. What we need is to put a lock into our email communications door. A lock that can be as hard to pick as the user wants, and yet easy to use as the user wants it to be used. Spammers are thiefs -- they steal time and resources, they make us reject legitimate email. They cost a lot of money to all of us. They have not been and will not be deterred by law alone. There is also no world law and spammers are often hidding behind legitimate users that change all the time. We can't lock the spammers' doors everywhere, we have to lock our door at our house. BTW, to propose something simple, running code helps before any discussion. In a system as complex as email, however, one would have to be naive to even think about running code before running comments. I thank you all for the public discussion on this topic, through which I have learned a great deal, including people's first barriers to change.
Re: covert channel and noise -- was Re: proposal ...
Ed Gerck wrote: Dean Anderson wrote: It isn't the case that the spammer intended to send a message about the superbowl, but somehow noise altered the message to a solicitation on viagra. Rather, they intended to send a message on viagra, and you recieved their message, noise free. But seeing the solication for viagra, you became upset, and reported a complaint about the inappropriate use of the channel. In information-theory-speak, you report a communication in violation of the security model; a covert or sneaky channel. I guess we agree that if the message can be read by the intended recipient then it's not in a covert channel. A covert channel is one that can't even be detected by the intended recipient. But, you may ask, who is the intended recipient? In anti-spam email systems, we can usually recognize three whos: #1: My MTA #2: My MUA #3: Myself The spammer's strategy is to send the spam message in such a way that it is undetectable by my MTA-MUA, while it is detectable and readable by me. In short, it needs to use a covert channel through my MTA-MUA, but not to me. An example of such a covert channel is if a spammer hides information in the subject line by using wrongly spelled forms of viagra, information which my MTA-MUA cannot detect. It's not a covert channel for me but it is for my defenses. But, once that message passes through to me, it becomes detectable, readable and I call it spam. Does this mean that spam is defined by the rule I (only) Know It When I See It! and we have to accept a large failure rate in preventing spam, that can only be solved by laws and law enforcement? I want to emphasize that it does not have to be. Suppose a user can make email senders pay a burden at their MTA or even at their MUA (e.g., a bounce requesting encryption and solution to a puzzle). In addition, using a selective scale defined by the user (so that selected mail senders have less burden) at their MTA and MUA, the user can make the spammer pay a price *as high as desired* by the user (not limited by R. Brown's comments). In such case, the covert channel cannot even be established unless the spammer pays a price -- a price that can be *as high as desired* by the *user*. This is the essence of the proposal (the devil is in details). I take the example of the front door of your house. If you leave it open, so that a thief has no burden getting in, a thief probably will steal something from you -- even though the law says that theft is illegal. What we need is to put a lock into our email communications door. A lock that can be as hard to pick as the user wants, and yet easy to use as the user wants it to be used. Spammers are thiefs -- they steal time and resources, they make us reject legitimate email. They cost a lot of money to all of us. They have not been and will not be deterred by law alone. There is also no world law and spammers are often hidding behind legitimate users that change all the time. We can't lock the spammers' doors everywhere, we have to lock our door at our house. BTW, to propose something simple, running code helps before any discussion. In a system as complex as email, however, one would have to be naive to even think about running code before running comments. I thank you all for the public discussion on this topic, through which I have learned a great deal, including people's first barriers to change.
Re: covert channel and noise -- was Re: proposal ...
[resending due to formatting error in previous msg] Dean Anderson wrote: It isn't the case that the spammer intended to send a message about the superbowl, but somehow noise altered the message to a solicitation on viagra. Rather, they intended to send a message on viagra, and you recieved their message, noise free. But seeing the solication for viagra, you became upset, and reported a complaint about the inappropriate use of the channel. In information-theory-speak, you report a communication in violation of the security model; a covert or sneaky channel. I guess we agree that if the message can be read by the intended recipient then it's not in a covert channel. A covert channel is one that can't even be detected by the intended recipient. But, you may ask, who is the intended recipient? In anti-spam email systems, we can usually recognize three whos: #1: My MTA #2: My MUA #3: Myself The spammer's strategy is to send the spam message in such a way that it is undetectable by my MTA-MUA, while it is detectable and readable by me. In short, it needs to use a covert channel through my MTA-MUA, but not to me. An example of such a covert channel is if a spammer hides information in the subject line by using wrongly spelled forms of viagra, information which my MTA-MUA cannot detect. It's not a covert channel for me but it is for my defenses. But, once that message passes through to me, it becomes detectable, readable and I call it spam. Does this mean that spam is defined by the rule I (only) Know It When I See It! and we have to accept a large failure rate in preventing spam, that can only be solved by laws and law enforcement? I want to emphasize that it does not have to be. Suppose a user can make email senders pay a burden at their MTA or even at their MUA (e.g., a bounce requesting encryption and solution to a puzzle). In addition, using a selective scale defined by the user (so that selected mail senders have less burden) at their MTA and MUA, the user can make the spammer pay a price *as high as desired* by the user (not limited by R. Brown's comments). In such case, the covert channel cannot even be established unless the spammer pays a price -- a price that can be *as high as desired* by the *user*. This is the essence of the proposal (the devil is in details). I take the example of the front door of your house. If you leave it open, so that a thief has no burden getting in, a thief probably will steal something from you -- even though the law says that theft is illegal. What we need is to put a lock into our email communications door. A lock that can be as hard to pick as the user wants, and yet easy to use as the user wants it to be used. Spammers are thiefs -- they steal time and resources, they make us reject legitimate email. They cost a lot of money to all of us. They have not been and will not be deterred by law alone. There is also no world law and spammers are often hidding behind legitimate users that change all the time. We can't lock the spammers' doors everywhere, we have to lock our door at our house. BTW, to propose something simple, running code helps before any discussion. In a system as complex as email, however, one would have to be naive to even think about running code before running comments. I thank you all for the public discussion on this topic, through which I have learned a great deal, including people's first barriers to change.
Re: covert channel and noise -- was Re: proposal ...
On Sun, 15 Feb 2004, Ed Gerck wrote: I take the example of the front door of your house. If you leave it open, so that a thief has no burden getting in, a thief probably will steal something from you -- even though the law says that theft is illegal. What we need is to put a lock into our email communications door. A lock that can be as hard to pick as the user wants, and yet easy to use as the user wants it to be used. No, this is an incorrect analogy. You want to put a lock on your mailbox. If you put a lock on your mailbox, you might as well not have one. The postman will no longer be able to deliver mail. And the notion that there will be more than one class of mail so that people will be able to send you postcards but not real mail is spurious. Unless you go through your postcard-class mail, you won't know if there is anything important in there. If you go through your postcard class mail, you have to visually filter and reject all of the spam and other unwanted mail that might be in it. I have this problem now as it is. I use spam assassin and set the spam level pretty high -- 5 -- and bounce all of the rejects into a file where I COULD check it. However, I get roughly 5 MB/week in this file so I don't, as a rule. Because I set it so high, VERY FEW legitimate messages are lost, but a fair bit of spam gets through, including various classes of stealth spam with [EMAIL PROTECTED] lines (note that just using this line in this email will guarantee its unviewed rejection by certain silly clients with much LOWER spam thresholds, set so that they dump a lot of real mail along with the spam). This is a personal choice -- the lower the threshold the more I reject and the greater the chance that real messages will be rejected along with the spam. If I have to review the rejected spam I lose -- I might as well not reject at all. If I lose an important message I also lose. THIS IS NOT A MATTER OF PROTOCOL! This is not the business of the IETF! Tools exist to help you filter spam. Some, like spam assassin, are very good and quite sophisticated. However, there is ALWAYS a tradeoff with using this sort of tool -- too narrow a criterion for acceptance and you lose real mail, too broad and you get all the spam anyway. I can choose my level of tradeoff, and be fully aware of the risks. Spammers are thiefs -- they steal time and resources, they make us reject legitimate email. They cost a lot of money to all of us. They have not been and will not be deterred by law alone. There is also no world law and spammers are often hidding behind legitimate users that change all the time. We can't lock the spammers' doors everywhere, we have to lock our door at our house. No, what we can do is the same thing we do with our real mail box. Make it illegal to send certain classes of mail, for example letter bombs and envelopes containing anthrax, and prosecute the hell out of anybody we catch who does so. We cannot arrange it so that whoever sends us mail that for any reason we don't want to accept has to solve a puzzle of indeterminate difficulty in order to send it to us. And no, I don't WANT to solve a puzzle (I presume, a human level puzzle) in order to send somebody email. I WON'T solve a puzzle to send somebody email -- anybody that silly will just not get email from me. I've already pointed out that any extra step involving e.g. encryption or a machine solveable puzzle is trivially manageable and trivially automateable at effectively zero cost to any spammer and backed it up with numbers -- cycles are available by the billion (literally) -- millions of cycles can be burned PER LETTER of the message and it's no skin off a spammer's bottom line. I reiterate -- most of what you propose is possible now. If you want to send only encrypted mail, you can. If you insist on signing all mail, you can. If you want to only accept signed, encrypted mail, you can. What you can't do is make everybody else go to all of that effort for no benefit that could even THEORETICALLY ensue. If you make it easy for everybody to send signed, encrypted mail, you make it easy for spammers to send signed, encrypted mail and you're back where you started. If you don't, then your cure is worse than the disease. BTW, to propose something simple, running code helps before any discussion. In a system as complex as email, however, one would have to be naive to even think about running code before running comments. I thank you all for the public discussion on this topic, through which I have learned a great deal, including people's first barriers to change. This is true only if you want to LISTEN to what other people say. What is at issue here isn't my barrier to change. It is that this is (in my opinion) a foolish idea. I get just as irritated at spam as the next person (more so, since as I noted I get some 5 MB a week that is rejected by my filters of spam and identifiable viruses combined).
