Re: master's mail backlog and upgrade time
On Thu, Nov 24, 2005 at 06:45:58PM +, Ian Jackson wrote: So the best idea is indeed for downstream systems to have policies which are no more strict than upstream systems. Would it be possible for master to make call-outs to chiark ? would that solve the problem ? Regards, Paddy -- Perl 6 will give you the big knob. -- Larry Wall -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
On Thu, Nov 24, 2005 at 06:45:58PM +, Ian Jackson wrote: So the best idea is indeed for downstream systems to have policies which are no more strict than upstream systems. Would it be possible for master to make call-outs to chiark ? would that solve the problem ? I don't think so. The rejections are mostly content-based, I guess. And DATA callouts are not really possible. There are two sane approaches here: block unwanted mail at master (especially for @debian.org mailbox you don't want to exist), and accept everything that does get through (only flagging spam, not rejecting it). I'm not sure if the first option has been implemented in the meantime. It shouldn't be too to do, now that master is running Exim 4, and once the desired semantics are clear. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
This one time, at band camp, Florian Weimer said: On Thu, Nov 24, 2005 at 06:45:58PM +, Ian Jackson wrote: So the best idea is indeed for downstream systems to have policies which are no more strict than upstream systems. Would it be possible for master to make call-outs to chiark ? would that solve the problem ? I don't think so. The rejections are mostly content-based, I guess. And DATA callouts are not really possible. There are two sane approaches here: block unwanted mail at master (especially for @debian.org mailbox you don't want to exist), and accept everything that does get through (only flagging spam, not rejecting it). I'm not sure if the first option has been implemented in the meantime. It shouldn't be too to do, now that master is running Exim 4, and once the desired semantics are clear. Since allow_fail is set on the ldap_aliases router, it seems that all we need is some way of letting users set their forward address to :fail: instead of an email address. I suppose the same scripts that let you reset your forwarding address on db.debian.org could also have some interace that made the address lookup fail. Take care, -- - | ,''`.Stephen Gran | | : :' :[EMAIL PROTECTED] | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
Stephen Frost writes (Re: master's mail backlog and upgrade time): *I* don't bounce much of anything. Talk to Ian about wanting to generate bounces, it wasn't my idea. What I want is for him to bounce it himself if he feels it needs to be bounced, not make master do it. What I want is for master not to accept the mail either. Even with better SPAM checking on master it's very unlikely that master's policy will ever agree completely with Ian's (and everyone else's, whose policies are probably different from Ian's..) and so this kind of setup is unlikely to ever actually work (where you're depending on your forwarding hosts to implement the same policy you have). In the modern internet you're always going to have some kind of breakage due to this kind of policy mismatch. I'm familiar with this problem myself because a significant proportion of my users use chiark (with its strict spamfiltering) as a published email address and then forward the email somewhere else; sometimes the somewhere else spots a problem with a mail that chiark didn't spot and then I usually end up dealing with bounced bounces. So the best idea is indeed for downstream systems to have policies which are no more strict than upstream systems. But to elevate this to a _requirement_ on the users whose mail is being forwarded presupposes that those users have a say in what that mail flow is. I don't have an adequate say in the mail arriving for me from master. There is no good reason why master should _ever_ accept mail for [EMAIL PROTECTED] from foreign systems. I think `I would like this mail flow to go away completely' is a perfectly reasonable position on the part of the user - at least, when the user is a Debian developer and the mail flow is that directed from strangers to their Debian account. And if the upstream system can't implement that policy then it's reasonable of the user to implement it themselves. After we've got that far there are three things that the user can have their system do to implement such a policy: 1. reject the mail at SMTP time; 2. accept the mail, make a best effort at sending a bounce, and if that fails discard the bounce; or 3. accept the mail and silently discard it. There are arguments in favour and against all of these options. None of them are good but it is unreasonable to /demand/ that the user make a particular choice. As it happens my arrangements currently imply a mixture of 1 and 2 depending on the type of unwanted mail, with a guesswork arrangement which attempts to show me mails which were generated _on_ the Debian systems or by Debian sysadmins. But that is a decision for me to take, surely. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
* Brian May ([EMAIL PROTECTED]) wrote: Are you saying you should bounce SPAM mail??? *I* don't bounce much of anything. Talk to Ian about wanting to generate bounces, it wasn't my idea. What I want is for him to bounce it himself if he feels it needs to be bounced, not make master do it. No, I don't think trying to enforce his policies on master is an option either, sorry. The real point here is that, for mail coming from master, it's *going* to get bounced one way or another, unless Ian decides to try to classify the message himself as 'deserving a bounce' or not. Yes, in this case the mail would bounce anyway, but I think the solution is to improve the SPAM checking on master, and not accept the mail in the first place. Even with better SPAM checking on master it's very unlikely that master's policy will ever agree completely with Ian's (and everyone else's, whose policies are probably different from Ian's..) and so this kind of setup is unlikely to ever actually work (where you're depending on your forwarding hosts to implement the same policy you have). Yes, you could probably make mail from master.debian.org bypass SMTP level SPAM controls, but taken to the extreme, you would have to also do this to every server that forwards you email (perhaps even every mailing list server?). That would be the point, yes. Taken a bit further, you'd have to clasify the mail as SPAM or not yourself and generate a bounce or not as appropriate. It's honestly not all that difficult. Thanks, Stephen signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
On Sat, Nov 19, 2005 at 01:56:25PM -0500, Stephen Frost wrote: * Ian Jackson ([EMAIL PROTECTED]) wrote: So if I have my system say `250' to a piece of mail, I'm guaranteeing that either I'll bounce it (and get a `250' on the bounce), or that some human (me or someone else I know) will read it. Sure, so say '250' and then bounce it if you want to later. That's basically the *point* here. master's forwarding the mail for you, you should accept it No, that's not what you should do. I don't want to accept any random crap that a forwarding host might send me just because I asked it to forward mail for me; my resources (in the form of bandwidth, processing time, and disk space) are limited, and if a mail is more than obviously not legitimate (i.e., because the envelope from doesn't even exist), then *I* don't want to spend more CPU time, disk space, or bandwidth handling the message; I'll just say thanks, but no thanks. If that creates problems for master, and there are ways to avoid those problems (other than by creating more problems for my systems) then I'll happily implement those; but you won't force me to accept any random crap you might send me just because your policies aren't strict enough. To push your example to the limit, you're suggesting that I should accept whatever master sends me -- even if that would result in it sending a constant stream of data to my home server of, say, 100Kbit/s. To accomodate for such an amount of data, however, I'd need to upgrade my mail server's disk, processor, and my home Internet connection -- which I'm not willing to do. Granted, the above example is over the top; but it doesn't even have to be that extreme. I'm currently nearly out of disk space on my main mail server; the extra amount of mail that dropping SMTP-time checks would result in would probably make my mailserver's disk overflow--and I don't want to buy another hard disk just yet. and then you can decide to either read it yourself or bounce it. You do *not* need master to bounce it for you! If master accepts a mail, it should be prepared to bounce it if the mail wasn't legitimate after all. If it isn't prepared to do so, it shouldn't have accepted it in the first place! The only practical solution to this problem in the modern environment is to never accept mail that you don't want. Unfortunately master's policies make it impossible for me to arrange to do that. I can do what I can, though, and try to push the problem closer to the place where it can be solved. Blacklisting obviously has its own problems. There are many more ways to rejecting mail at SMTP time than to start blacklisting; additionally, provided one uses a serious blacklist, blacklisting isn't likely to result in blocking one's own forwarding host. -- .../ -/ ---/ .--./ / .--/ .-/ .../ -/ ../ -./ --./ / -.--/ ---/ ..-/ .-./ / -/ ../ --/ ./ / .--/ ../ -/ / / -../ ./ -.-./ ---/ -../ ../ -./ --./ / --/ -.--/ / .../ ../ --./ -./ .-/ -/ ..-/ .-./ ./ .-.-.-/ / --/ ---/ .-./ .../ ./ / ../ .../ / ---/ ..-/ -/ -../ .-/ -/ ./ -../ / -/ ./ -.-./ / -./ ---/ .-../ ---/ --./ -.--/ / .-/ -./ -.--/ .--/ .-/ -.--/ .-.-.-/ / ...-.-/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
James Troup writes (Re: master's mail backlog and upgrade time): The change was made roughly less than 24 hours before your first post to debian-devel. There wasn't actually all that much time to contact you in. You (plural) could have _just_ contacted me and I would have fixed it, as I have now done. It's not like you don't know how to get hold of me ! How hard is it to write a short mail ? `master is having difficulty talking to chiark; chiark seems to have firewalled master out. Please fix it ASAP'. You would have had it sorted out, with apology, straight away. Err, no you haven't [arranged matters]. In [EMAIL PROTECTED] posted on Sat, 19 Nov 2005 15:39:36 +, you've asked chiark users to individually change their config to ensure chiark won't DoS master. Until it's confirmed that all those users have done so, it's not fair to say master will not have any difficulty talking to chiark. The problem with chiark becoming upset at master will not recur so long as a majority of the traffic is to users who have been thus reconfigured. This is now the case (and I'm chasing up the few stragglers). And this still relies on per-user config rather than being a global change. Which means the next time a chiark user gets a debian.org account but neglects to perform the necessary incantation, master will once again be DoSed by chiark. Doing it this way means that if and when the debian.org hosting arrangments change (eg, the host that talks to chiark gets a different reverse DNS name) the configuration will still work and it won't break again. It's only old forwarding arrangements that should have the problem - new chiark users get clear instructions on what to do about mail forwarding, and I'm sure that new debian.org users do too. I also immediately notified debian-admin of these facts and have not had: Err, no you didn't. You mailed debian-devel, and THREE DAYS LATER mailed debian-admin. That is not immediate. I replied to Ryan directly, in response to his posting to d-d-a. There was no other contact address given in his announcement. * Correction of the broken configuration on master [EMAIL PROTECTED]:~$ grep chiark /etc/exim4/exim4.conf [EMAIL PROTECTED]:~$ Oh, thanks. And it's been like that for several days. Oh! Nice of someone to tell me so that I could watch my logs and check that all was well. As it turns out it wasn't because I had underestimated the number of other users and the volume of their spam flows. Speaking of doing better, could you please actually fix chiark? I should apologise to you for causing trouble, of course. So yes, I am sorry that my system didn't like your system. Because it's STILL firewalling master As I say, I've chased up some stragglers this morning about their configuration and now the vast majority of the spam flow is marked so that chiark likes getting crap, rather than getting annoyed about it. This seem to be working as far as I can tell from here. This situation is intolerable and must be rectified forthwith. With all due respect, your attitude is intolerable and should be rectified forthwith. I _am_ sorry to be such a pain about this but escalation seemed to be the only way to get a response ! Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
* Wouter Verhelst ([EMAIL PROTECTED]) wrote: I don't want to accept any random crap that a forwarding host might send me just because I asked it to forward mail for me; my resources (in the form of bandwidth, processing time, and disk space) are limited, and if Then don't run a mail server. Thanks, Stephen signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
On Tue, Nov 22, 2005 at 10:57:27AM -0500, Stephen Frost wrote: * Wouter Verhelst ([EMAIL PROTECTED]) wrote: I don't want to accept any random crap that a forwarding host might send me just because I asked it to forward mail for me; my resources (in the form of bandwidth, processing time, and disk space) are limited, and if Then don't run a mail server. That's not the way things work. My resources are _always_ limited, even if I'd have a mailserver built out of the latest-and-greatest hardware (which, granted, is not the case, but still). It's unconsiderate for _any_ mail system to be sending mail to me of which it's blindingly obvious that I wouldn't want to have it; and I'll just reject it if that's the case. -- .../ -/ ---/ .--./ / .--/ .-/ .../ -/ ../ -./ --./ / -.--/ ---/ ..-/ .-./ / -/ ../ --/ ./ / .--/ ../ -/ / / -../ ./ -.-./ ---/ -../ ../ -./ --./ / --/ -.--/ / .../ ../ --./ -./ .-/ -/ ..-/ .-./ ./ .-.-.-/ / --/ ---/ .-./ .../ ./ / ../ .../ / ---/ ..-/ -/ -../ .-/ -/ ./ -../ / -/ ./ -.-./ / -./ ---/ .-../ ---/ --./ -.--/ / .-/ -./ -.--/ .--/ .-/ -.--/ .-.-.-/ / ...-.-/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
I demand that Simon Richter may or may not have written... Rolf Kutz wrote: emails because of obviously nonexistent envelope addresses, that doesn't count those systems where we don't accept mail from *at all* because they are dialup systems. This, however, is a small system with 10 email How do you define dialup systems and tell dialup systems from other systems? There is a database where ISPs can register the ranges they assign for dialup users. Isn't that for dynamic-IP dial-up only? -- | Darren Salt | linux (or ds) at | nr. Ashington, | sarge,| youmustbejoking | Northumberland | RISC OS | demon co uk | Toon Army | I don't ask for much, just untold riches... I am Spock of Borg. Resistance is illogical. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Hi, Darren Salt wrote: There is a database where ISPs can register the ranges they assign for dialup users. Isn't that for dynamic-IP dial-up only? AFAIK there are two lists, however only few static dialup IPs are registered -- after all, the interesting attribute is whether the addresses are dynamically assigned, not whether the network connection is intermittent (mail servers are supposed to handle that as a temporary error, while talking to the wrong server will lead to a 5xx and the mail bouncing). Simon signature.asc Description: OpenPGP digital signature
Re: master's mail backlog and upgrade time
Stephen == Stephen Frost [EMAIL PROTECTED] writes: So if I have my system say `250' to a piece of mail, I'm guaranteeing that either I'll bounce it (and get a `250' on the bounce), or that some human (me or someone else I know) will read it. Stephen Sure, so say '250' and then bounce it if you want to Stephen later. That's basically the *point* here. master's Stephen forwarding the mail for you, you should accept it and Stephen then you can decide to either read it yourself or bounce Stephen it. You do *not* need master to bounce it for you! Are you saying you should bounce SPAM mail??? Where do you bounce it to? The forged senders email address? Alternatively, you could redirect the mail to /dev/null, but then the poster of a genuine email doesn't know his mail didn't get through. Yes, in this case the mail would bounce anyway, but I think the solution is to improve the SPAM checking on master, and not accept the mail in the first place. Yes, you could probably make mail from master.debian.org bypass SMTP level SPAM controls, but taken to the extreme, you would have to also do this to every server that forwards you email (perhaps even every mailing list server?). -- Brian May [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Hi, Andreas Metzler wrote: The real problem with these bounces is not that they fill up the forwarding host's queue but that they are usually unwanted. Think Joe Job. This thread is about email that is obviously not legitimate just looking at the envelope. In this day and age, everyone does ingress/egress filtering on their networks to enforce just that minimum bit of plausibility, and I feel email systems should do the same. In the last one and a half days a system I administer has rejected 451 emails because of obviously nonexistent envelope addresses, that doesn't count those systems where we don't accept mail from *at all* because they are dialup systems. This, however, is a small system with 10 email addresses total. If legitimate email is rejected because the envelope is obviously broken, I believe it is rightfully so, and the sender of that email is supposed to do something about it. The correct behaviour to notify the sender in this case is to reject the mail at the SMTP level, because this is going to be the last time you have a connection to someone who is responsible in some way or other (either by sending broken emails or by running an email server accepting broken emails). Forwarding email unconditionally makes my debian.org address receive by far the largest amount of spam of all addresses I have. Please, think about the kittens. Simon signature.asc Description: OpenPGP digital signature
Re: master's mail backlog and upgrade time
* Quoting Simon Richter ([EMAIL PROTECTED]): emails because of obviously nonexistent envelope addresses, that doesn't count those systems where we don't accept mail from *at all* because they are dialup systems. This, however, is a small system with 10 email How do you define dialup systems and tell dialup systems from other systems? - Rolf -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Hello, Rolf Kutz wrote: emails because of obviously nonexistent envelope addresses, that doesn't count those systems where we don't accept mail from *at all* because they are dialup systems. This, however, is a small system with 10 email How do you define dialup systems and tell dialup systems from other systems? There is a database where ISPs can register the ranges they assign for dialup users. Simon signature.asc Description: OpenPGP digital signature
Re: master's mail backlog and upgrade time
Six days ago I discovered that one of the Debian system administrators had made a deliberate and highly unusual configuration change which predictably broke mail from or via master to: * me personally * some of the =8 other Debian developers who have accounts on chiark * the Technical Committee No-one had contacted me, or anyone else affected. The exact causes and responsibility for the root problem that the administrator was trying to solve have been discussed extensively in the thread on debian-devel, but in any case immediately I found out about the situation I arranged matters so that master should no longer have any difficulty talking to chiark. I also immediately notified debian-admin of these facts and have not had: * Correction of the broken configuration on master * An apology for the lack of communication or a promise to do better This situation is intolerable and must be rectified forthwith. If the Debian system administrators are too busy to deal with this then I would be happy to help. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Simon Richter [EMAIL PROTECTED] wrote: Andreas Metzler wrote: The real problem with these bounces is not that they fill up the forwarding host's queue but that they are usually unwanted. Think Joe Job. This thread is about email that is obviously not legitimate just looking at the envelope. [...] I missed that piece if information. Thanks for pointing it out. cu andreas -- The 'Galactic Cleaning' policy undertaken by Emperor Zhark is a personal vision of the emperor's, and its inclusion in this work does not constitute tacit approval by the author or the publisher for any such projects, howsoever undertaken.(c) Jasper Ffforde -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Ian Jackson [EMAIL PROTECTED] writes: Six days ago I discovered that one of the Debian system administrators had made a deliberate and highly unusual configuration change which predictably broke mail from or via master to: Err, no. Mail was _already_ bouncing, but after reaching the retry maximum. The change did not change the end result, only when it happened. No-one had contacted me, or anyone else affected. The change was made roughly less than 24 hours before your first post to debian-devel. There wasn't actually all that much time to contact you in. but in any case immediately I found out about the situation I arranged matters so that master should no longer have any difficulty talking to chiark. Err, no you haven't. In [EMAIL PROTECTED] posted on Sat, 19 Nov 2005 15:39:36 +, you've asked chiark users to individually change their config to ensure chiark won't DoS master. Until it's confirmed that all those users have done so, it's not fair to say master will not have any difficulty talking to chiark. And this still relies on per-user config rather than being a global change. Which means the next time a chiark user gets a debian.org account but neglects to perform the necessary incantation, master will once again be DoSed by chiark. I also immediately notified debian-admin of these facts and have not had: Err, no you didn't. You mailed debian-devel, and THREE DAYS LATER mailed debian-admin. That is not immediate. * Correction of the broken configuration on master [EMAIL PROTECTED]:~$ grep chiark /etc/exim4/exim4.conf [EMAIL PROTECTED]:~$ And it's been like that for several days. * An apology for the lack of communication or a promise to do better Speaking of doing better, could you please actually fix chiark? Because it's STILL firewalling master This situation is intolerable and must be rectified forthwith. With all due respect, your attitude is intolerable and should be rectified forthwith. If the Debian system administrators are too busy to deal with this then I would be happy to help. No thanks. -- James -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Andy Smith [EMAIL PROTECTED] wrote: On Fri, Nov 18, 2005 at 12:48:30PM -0500, Stephen Frost wrote: * Ian Jackson ([EMAIL PROTECTED]) wrote: Stephen Frost writes (Re: master's mail backlog and upgrade time): Then bounce it locally. Duh. No reason to force master to deal with the bounce messages you feel are 'right' to send. I don't bounce it. I reject it at SMTP time with a 4xx or 5xx code. Congradulations! You've found the problem! You would prefer that Ian: a) inflict bounce spam scatter on the forged from addresses in the malware and spam he doesn't want to accept delivery for; or b) silently discards such mails resulting in the possibility of legitimate mail being lost; or c) just accepts the spam/malware? [...] Hello, FWIW he currently does a. Rejecting at SMTP time causes backscatter on forwarded mail, as the forwarding host cannot reject because it already has accepted the mail. For my personal mail I usually reject spam that is directly delivered and drop the one I receive through forwarding. - Which is easy for me as this is just my personal mail and I know which hosts forward to me. - I guess this is next to impossile for Ian. cu andreas -- The 'Galactic Cleaning' policy undertaken by Emperor Zhark is a personal vision of the emperor's, and its inclusion in this work does not constitute tacit approval by the author or the publisher for any such projects, howsoever undertaken.(c) Jasper Ffforde -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
2005/11/19, Andreas Metzler [EMAIL PROTECTED]: FWIW he currently does a. Rejecting at SMTP time causes backscatter on forwarded mail, as the forwarding host cannot reject because it already has accepted the mail. And usual way to deal with this is to set: ignore_errmsg_errors_after = 7d If a bounce message can't be delivered they are frozen. After 7 days they are deleted. Problem solved. I can't think of a reason that the mail would keep building up... Have a nice day,
Re: master's mail backlog and upgrade time
Andy Smith writes (Re: master's mail backlog and upgrade time): Instead of either side in this debate saying Not my problem, you should do this... how about reaching some compromise? It sounds like in the short term, Ian needs to discard some mail instead of rejecting, and in the long term master needs to be able to cope with this sort of thing. The absolute worst thing to do is to start generating bounces to these forged addresses however. I would like to point out that the real original operational problem here was _not_ that chiark was making master bounce junkmail. The real problem was that the mail from from master to chiark consisted so overwhelmingly of junk that chiark's teergrube kicked in. The teergrube is _designed_ to soak up capacity from spamming sites. It is of course a problem when applied to `friendly' mail flows[1] and that's why I have now told chiark not to do that. That the forwarded mail triggered the teergrube at all was due to the fact that my debian.org mail forwarding arrangements predate the Internet mail climate requiring the ability to selectively disable the teergrube for forwarded mail. So, I and some of my users' mail didn't have that feature turned on, which is why master got bogged down. It is of course a bug in master's mail system that one unfriendly host can sap all of its capacity, but even without that bug there would still have been a problem because _wanted_ mail via chiark wouldn't have got through. This kind of thing (warring defensive systems) is going to happen occasionally in the current mail environment. The important thing is that when you see a problem with a supposedly-friendly system you should talk to someone to get it fixed ! And, of course, that people take a flexible attitude towards fixing things. Otherwise we'll all be doomed. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Stephen Frost writes (Re: master's mail backlog and upgrade time): * Andy Smith ([EMAIL PROTECTED]) wrote: a) inflict bounce spam scatter on the forged from addresses in the malware and spam he doesn't want to accept delivery for; or ... It's his choice to do either (a) or (b) or (c). I couldn't care less which he does provided *he* does it. I do *not* want him to make master do (a) for him. The problem with no-one sending bounces is that _legitimate_ mail which is _mistakenly_ tagged as spam just vanishes. If we want to have _reliable_ mail, ie mail which doesn't ever just `vanish', we _must_, after accepting a mail, either deliver it, or bounce it. If a system can't do either of those then the human system administrator has to read them, or forward them to some other human who will read them. That's my reading of RFC1123 section 5.3.3, and would have been everyone else's before mass market ISPs started throwing away bounced bounces en masse. So if I have my system say `250' to a piece of mail, I'm guaranteeing that either I'll bounce it (and get a `250' on the bounce), or that some human (me or someone else I know) will read it. The only practical solution to this problem in the modern environment is to never accept mail that you don't want. Unfortunately master's policies make it impossible for me to arrange to do that. I can do what I can, though, and try to push the problem closer to the place where it can be solved. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Martijn van Oosterhout [EMAIL PROTECTED] wrote: 2005/11/19, Andreas Metzler [EMAIL PROTECTED]: FWIW he currently does a. Rejecting at SMTP time causes backscatter on forwarded mail, as the forwarding host cannot reject because it already has accepted the mail. And usual way to deal with this is to set: ignore_errmsg_errors_after = 7d If a bounce message can't be delivered they are frozen. After 7 days they are deleted. Problem solved. I can't think of a reason that the mail would keep building up... Hello, The real problem with these bounces is not that they fill up the forwarding host's queue but that they are usually unwanted. Think Joe Job. cu andreas -- The 'Galactic Cleaning' policy undertaken by Emperor Zhark is a personal vision of the emperor's, and its inclusion in this work does not constitute tacit approval by the author or the publisher for any such projects, howsoever undertaken.(c) Jasper Ffforde -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
* Steinar H. Gunderson ([EMAIL PROTECTED]) wrote: On Fri, Nov 18, 2005 at 02:11:43PM -0500, Stephen Frost wrote: I expect you could do it though I havn't tried myself because I'm not a big fan of smtp-level rejects exactly for these reasons. I just accept and then discard (at least for known userids, but I don't expect many people to be setting up forwards for non-existant userids). What do you do when you encounter a false positive? Not everybody has the luxury of affording to have their legitimate mail eaten silently. This is rather orthogonal to the issue at hand, but since you asked, I don't do anything about false positives. If I learn about it then I'll adjust my rules. I don't think it makes sense to create a bounce for every spam that comes my way though. Thanks, Stephen signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
* Ian Jackson ([EMAIL PROTECTED]) wrote: Stephen Frost writes (Re: master's mail backlog and upgrade time): * Andy Smith ([EMAIL PROTECTED]) wrote: a) inflict bounce spam scatter on the forged from addresses in the malware and spam he doesn't want to accept delivery for; or ... It's his choice to do either (a) or (b) or (c). I couldn't care less which he does provided *he* does it. I do *not* want him to make master do (a) for him. The problem with no-one sending bounces is that _legitimate_ mail which is _mistakenly_ tagged as spam just vanishes. Shrug Life's a beach. I don't think creating huge numbers of bounces is better than this. Make no mistake, that's what you're doing, even though you're making other systems do it for you it's your fault they're happening. So if I have my system say `250' to a piece of mail, I'm guaranteeing that either I'll bounce it (and get a `250' on the bounce), or that some human (me or someone else I know) will read it. Sure, so say '250' and then bounce it if you want to later. That's basically the *point* here. master's forwarding the mail for you, you should accept it and then you can decide to either read it yourself or bounce it. You do *not* need master to bounce it for you! The only practical solution to this problem in the modern environment is to never accept mail that you don't want. Unfortunately master's policies make it impossible for me to arrange to do that. I can do what I can, though, and try to push the problem closer to the place where it can be solved. Blacklisting obviously has its own problems. It's your choice to do it but don't do it to hosts who are forwarding mail to you. If you can't figure out which hosts are forwarding to you and which aren't then either don't blacklist or don't run a mail server. Thanks, Stephen signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
Steve Langasek writes (Re: master's mail backlog and upgrade time): On Tue, Nov 15, 2005 at 04:01:10PM +, Ian Jackson wrote: But, there is another important point: I don't really want a debian.org address. It's technically necessary for me to have one for (eg) cronmail from debian systems, and additionally I feel that there is an (unwritten) rule that I should have one. But it is simply untenable to suggest that I ought to accept all of this junkmail and actually read it ! So accept it and auto-discard it instead, if you prefer; but don't throw it back at master after telling master to send it to you. I'm strange in that I like my mail to be reliable. In particular, I want people who try to mail me, and fail, to be told about it. This is unpopular in these days of dumbed-down, and even hidden, error messages. But I still think it's right. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Steve Langasek writes (Re: master's mail backlog and upgrade time): Anyway, the line in question is still in master's exim4 config; you may want to try sending a mail to debian-admin, let them know what you've done on your end, and ask if there's anything still preventing its removal... Well, I had sort of hoped that CCing my original response to Ryan - who did, after all, make the original announcement - would have had some useful effect. I've now mailed debian-admin - see attached. Ian. ---BeginMessage--- On debian-devel, CC Ryan Murray, I wrote: * I now discover by reading master's exim4.conf that all mail to the _mail domain_ chiark.greenend.org.uk has been arranged to bounce on master. This is contrary to what is stated in the announcement, which says that the setup was changed `to not deliver to the problem _address_' (emph. mine). I would like to draw your attention to this thread. Please respond, either in private or in public. Please correct the mail configuration on master. If you are too busy to do it then please give me the privilege necessary to do it myself. Thanks, Ian. ---End Message---
Re: master's mail backlog and upgrade time
* Ian Jackson ([EMAIL PROTECTED]) wrote: Steve Langasek writes (Re: master's mail backlog and upgrade time): So accept it and auto-discard it instead, if you prefer; but don't throw it back at master after telling master to send it to you. I'm strange in that I like my mail to be reliable. In particular, I want people who try to mail me, and fail, to be told about it. This is unpopular in these days of dumbed-down, and even hidden, error messages. But I still think it's right. Then bounce it locally. Duh. No reason to force master to deal with the bounce messages you feel are 'right' to send. Stephen signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
* Ian Jackson ([EMAIL PROTECTED]) wrote: Stephen Frost writes (Re: master's mail backlog and upgrade time): Then bounce it locally. Duh. No reason to force master to deal with the bounce messages you feel are 'right' to send. I don't bounce it. I reject it at SMTP time with a 4xx or 5xx code. Congradulations! You've found the problem! Thanks, Stephen signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
Stephen Frost writes (Re: master's mail backlog and upgrade time): Then bounce it locally. Duh. No reason to force master to deal with the bounce messages you feel are 'right' to send. I don't bounce it. I reject it at SMTP time with a 4xx or 5xx code. Iaan. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
On Fri, Nov 18, 2005 at 12:48:30PM -0500, Stephen Frost wrote: * Ian Jackson ([EMAIL PROTECTED]) wrote: Stephen Frost writes (Re: master's mail backlog and upgrade time): Then bounce it locally. Duh. No reason to force master to deal with the bounce messages you feel are 'right' to send. I don't bounce it. I reject it at SMTP time with a 4xx or 5xx code. Congradulations! You've found the problem! You would prefer that Ian: a) inflict bounce spam scatter on the forged from addresses in the malware and spam he doesn't want to accept delivery for; or b) silently discards such mails resulting in the possibility of legitimate mail being lost; or c) just accepts the spam/malware? I'm guessing (b), with the reasoning that if he chooses to reject mail that his system thinks is bad then it's his problem to deal with any false positives. However in this day and age of the unwanted ratio of email being greater than the wanted ratio, any system which accepts a lot of unwanted email and then fails to deal with the refusal to accept by systems further down the line is in real trouble. I do pretty much the same as what Ian does, as I have explained, and so do many others. It's the best way to deal with such mail: don't accept what you're not prepared to deal with. Instead of either side in this debate saying Not my problem, you should do this... how about reaching some compromise? It sounds like in the short term, Ian needs to discard some mail instead of rejecting, and in the long term master needs to be able to cope with this sort of thing. The absolute worst thing to do is to start generating bounces to these forged addresses however. My 2p, Andy signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
On Fri, 18 Nov 2005, Stephen Frost wrote: * Ian Jackson ([EMAIL PROTECTED]) wrote: Stephen Frost writes (Re: master's mail backlog and upgrade time): Then bounce it locally. Duh. No reason to force master to deal with the bounce messages you feel are 'right' to send. I don't bounce it. I reject it at SMTP time with a 4xx or 5xx code. Congradulations! You've found the problem! *THE* problem? *THE* problem is related to master's MTA configuration, handling and administration, and not in anything else that gives it extra work to do. Bouncing crap to master is *a* problem, because one should not be bouncing back stuff to middle-of-the-way MTAs that are known in one's setup, and master certainly qualifies. Since we are talking about it, it is not always trivial to special-case an incoming connection for a local bounce instead of a SMTP-level bounce, though. At least not with all MTAs. I can do so with postfix because I have a two-layer setup anyway (mail gets twice through the system, for content filtering with extremely high input rate. I just tell the first instance to let master through, and the second one to reject -- this causes a *local* bounce, as mail is already queued and accepted). I am not even sure if it can be done [in postfix] in a single layer setup (smtpd-queues-MDA). Anyone would know how to do it? -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
* Andy Smith ([EMAIL PROTECTED]) wrote: You would prefer that Ian: a) inflict bounce spam scatter on the forged from addresses in the malware and spam he doesn't want to accept delivery for; or That is what he's said he wants to do. What I want him to do is have *his* servers do it, not make master do it. b) silently discards such mails resulting in the possibility of legitimate mail being lost; or c) just accepts the spam/malware? I'm guessing (b), with the reasoning that if he chooses to reject mail that his system thinks is bad then it's his problem to deal with any false positives. It's his choice to do either (a) or (b) or (c). I couldn't care less which he does provided *he* does it. I do *not* want him to make master do (a) for him. However in this day and age of the unwanted ratio of email being greater than the wanted ratio, any system which accepts a lot of unwanted email and then fails to deal with the refusal to accept by systems further down the line is in real trouble. I do pretty much the same as what Ian does, as I have explained, and so do many others. It's the best way to deal with such mail: don't accept what you're not prepared to deal with. Don't do this to servers which are forwarding mail to you (upon request). It's inconsiderate, at best. Instead of either side in this debate saying Not my problem, you should do this... how about reaching some compromise? It sounds like in the short term, Ian needs to discard some mail instead of rejecting, and in the long term master needs to be able to cope with this sort of thing. The absolute worst thing to do is to start generating bounces to these forged addresses however. Erm, that's *exactly* what's happening today though, it's just that Ian's making master do it instead of doing it himself. Stephen signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
* Henrique de Moraes Holschuh ([EMAIL PROTECTED]) wrote: Since we are talking about it, it is not always trivial to special-case an incoming connection for a local bounce instead of a SMTP-level bounce, though. At least not with all MTAs. Using an MTA with the capabilities you need should be a prerequisite to running an MTA. I can do so with postfix because I have a two-layer setup anyway (mail gets twice through the system, for content filtering with extremely high input rate. I just tell the first instance to let master through, and the second one to reject -- this causes a *local* bounce, as mail is already queued and accepted). I am not even sure if it can be done [in postfix] in a single layer setup (smtpd-queues-MDA). Anyone would know how to do it? I expect you could do it though I havn't tried myself because I'm not a big fan of smtp-level rejects exactly for these reasons. I just accept and then discard (at least for known userids, but I don't expect many people to be setting up forwards for non-existant userids). I would have thought that at the least you could accept it, detemine it's SPAM and then have a procmail bounce-creating rule without all that much difficulty. Enjoy, Stephen signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
On Fri, Nov 18, 2005 at 02:11:43PM -0500, Stephen Frost wrote: I expect you could do it though I havn't tried myself because I'm not a big fan of smtp-level rejects exactly for these reasons. I just accept and then discard (at least for known userids, but I don't expect many people to be setting up forwards for non-existant userids). What do you do when you encounter a false positive? Not everybody has the luxury of affording to have their legitimate mail eaten silently. /* Steinar */ -- Homepage: http://www.sesse.net/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Steve Langasek [EMAIL PROTECTED] wrote: No: there is nothing proper about rejecting mail from a host that you have configured to forward mail for you. Nearly all of this mail flow is invalid in one way or another. Of course it is. That doesn't make it proper to reject such mail when you've told some other host to forward it to you in the first place; I'm sure it's a pretty common misconfiguration in the context of Debian mail forwarding, but that doesn't make it right... How can I avoid such a problem if I do not have the power to influence the mail server configuration of the machine that hosts the account I forward my Debian mail to? I currently forward Debian mail to [EMAIL PROTECTED], and although the domain is mine, the MX is not. I have no way to control the spam or any other policy of this server. Fortunately (or rather unfortunately except in this case) this server is really lax in accepting spam (they only offer to mark it in the subject). But if it was strict, what should I do? Not forward Debian mail and instead get it from master with POP3? Regards, Frank -- Frank Küster Inst. f. Biochemie der Univ. Zürich Debian Developer
Re: master's mail backlog and upgrade time
On Thu, Nov 17, 2005 at 01:41:31PM +0100, Frank Küster wrote: Steve Langasek [EMAIL PROTECTED] wrote: No: there is nothing proper about rejecting mail from a host that you have configured to forward mail for you. Nearly all of this mail flow is invalid in one way or another. Of course it is. That doesn't make it proper to reject such mail when you've told some other host to forward it to you in the first place; I'm sure it's a pretty common misconfiguration in the context of Debian mail forwarding, but that doesn't make it right... How can I avoid such a problem if I do not have the power to influence the mail server configuration of the machine that hosts the account I forward my Debian mail to? I currently forward Debian mail to [EMAIL PROTECTED], and although the domain is mine, the MX is not. I have no way to control the spam or any other policy of this server. Fortunately (or rather unfortunately except in this case) this server is really lax in accepting spam (they only offer to mark it in the subject). But if it was strict, what should I do? Not forward Debian mail and instead get it from master with POP3? Grabbing your mail off of master is one option. For that matter, continuing to forward it is an option, too... just, y'know, don't try to claim that sending SMTP rejects back to master is a *good* thing... -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. [EMAIL PROTECTED] http://www.debian.org/ signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15 Nov 2005, at 2:34 pm, Steve Langasek wrote: * The mail backlog that `will never be able to be delivered' was (as far as I can tell) all spam that chiark has been properly rejecting. No: there is nothing proper about rejecting mail from a host that you have configured to forward mail for you. I can see where you're coming from, but it's unavoidable, isn't it? Most of us probably have accounts which forward email to us, and over which we have no control; for example in my case I have two, one at debian.org, and one at Cambridge University (cantab.net), as well as any number of mailing lists. If those upstream sources are more lax about spam than the downstream SMTP receiver (whether chiark or something else) then this sort of thing is inevitable. I happen to have a chiark account, and my chiark address is the primary one I use for Debian development (see the maintainer field for any of my packages, or indeed the address from which I post to Debian mailing lists like this one) so I have some personal interest in this particular issue. It does seem it was a little hasty to have blanket-banned chiark- bound email, especially when it is well known that there are a significant number of DD's that use the machine. I specifically use chiark for things as publicly visible as my Debian package maintenance address, news and mailing list postings, precisely *because* of its somewhat draconian anti-spam measures. Tim -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iQEVAwUBQ3tHZRypeFo2odvPAQKRoggAtoiXmeOkePAFtBsP8LDpniinK9a88VEx OBrZtpk+yWmbMAX6y8Rifq2df62HDy4d2hTlBg26/bPXckhZiWhbIuJL8Ev1VxmI hggQ1knqgUyKCuiEXmuLO/ueH2wCN+mzc9coFN+Nu4cVp6QQuuYZAP4Yz8oIm3DO mFEw8N2lDRLJIbxg2RNHD71hkOtpHu9AGal1k0+GwDbLeVni4Wx4TxKXsRxD5bLV D1bruCihwtXJIhHK2CornOW9fljsOc8IipO/43Rt3tB+ks+3g89LPZQVcCu8ZbWK 7Bx2EheaWC0STqSpJRGqMTCH2oxnS0PfPFsRf/i3zMS5YX+usuzBbA== =9LTp -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
On Wed, Nov 16, 2005 at 02:51:10PM +, Tim Cutts wrote: On 15 Nov 2005, at 2:34 pm, Steve Langasek wrote: No: there is nothing proper about rejecting mail from a host that you have configured to forward mail for you. I can see where you're coming from, but it's unavoidable, isn't it? Most of us probably have accounts which forward email to us, and over which we have no control; for example in my case I have two, one at debian.org, and one at Cambridge University (cantab.net), as well as any number of mailing lists. If those upstream sources are more lax about spam than the downstream SMTP receiver (whether chiark or something else) then this sort of thing is inevitable.o Personally I have sa-exim SMTP reject viruses and high scoring spam, but if the message has precedence bulk or list then I discard it silently instead, in order to try and cut down on the number of sitations where my users will infuriate hosts that forward mail to them. Many MLMs will automatically kick you off the lists if you reject viruses they sent to you (hi, Phillipp Kern). It's not possible to catch everything though (users can set up forwards without my knowledge; they also can't predict which addresses will get lots of spam/malware), neither would I expect to be told I have to accept delivery of this email. In an ideal world both sides would be able to come to some arrangement that doesn't involve blackholing. signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
On Tue, Nov 15, 2005 at 04:01:10PM +, Ian Jackson wrote: If a domain was set up to be treated this way for an unrelated reasons without an announcement anywhere, surely that is even worse ! Well, it's no longer DSA is making misleading statements about the nature of the problem; the fact that you weren't notified when this was done is bad, but it could be that they tried to contact you and failed for whatever reason, so I certainly reserve any judgement until both sides have commented... On Tue, Nov 15, 2005 at 12:18:45PM +, Ian Jackson wrote: * The mail backlog that `will never be able to be delivered' was (as far as I can tell) all spam that chiark has been properly rejecting. No: there is nothing proper about rejecting mail from a host that you have configured to forward mail for you. Nearly all of this mail flow is invalid in one way or another. Of course it is. That doesn't make it proper to reject such mail when you've told some other host to forward it to you in the first place; I'm sure it's a pretty common misconfiguration in the context of Debian mail forwarding, but that doesn't make it right... What is happening is that master provides a mail forwarding facility with a lax input policy, but I forward my mail to chiark which does stricter checks and has a stricter policy. This has the effect of turning the rejected messages into bounced bounces on master. This would be an unfriendly thing to do if the sysadmins on master actually cared about reliable mail delivery and took a policy of reviewing bounced bounces and dealing with their causes (as I do on chiark). It's an unfriendly thing to do *anyway*. Performance of Debian mail service has sucked for a while, and based on my own experiences running such systems, I have no doubt that bounced forwards account for a lot of this sluggishness, not even counting the one particular user who was clogging the system with gigs of mail. Ryan now seems to be making quite a bit of progress on fixing these problems with the mail setup -- not just with eliminating antisocial mail forwarding configs, but also with making improvements on the input side so master doesn't *have* to store and forward so much garbage email. Like many others, my first reaction is Finally! Thank GOD!, but obviously DSA is comprised of volunteers like everything in Debian, and kudos are certainly due for his work on this - even if some of the changes may have been implemented in a suboptimal fashion. Anyway, the line in question is still in master's exim4 config; you may want to try sending a mail to debian-admin, let them know what you've done on your end, and ask if there's anything still preventing its removal... But, there is another important point: I don't really want a debian.org address. It's technically necessary for me to have one for (eg) cronmail from debian systems, and additionally I feel that there is an (unwritten) rule that I should have one. But it is simply untenable to suggest that I ought to accept all of this junkmail and actually read it ! So accept it and auto-discard it instead, if you prefer; but don't throw it back at master after telling master to send it to you. ... procmail? How would that help ? I could arrange for procmail to try to detect whether (eg) the mail had ever been through any non-debian.org host and if so construct a bounce. But this is no better and no worse than me having chiark rejecting the mails at SMTP time. It would still lead to master having to deal with lots of undeliverable bounces. Why would you need to bounce instead of discarding? If it's not a valid contact address for you, I don't see why you would be so concerned about sending notifications to people trying to use it. That was relevant 5-10 years ago; today it's a waste of resources. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. [EMAIL PROTECTED] http://www.debian.org/ signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
Ryan Murray writes (master's mail backlog and upgrade time): Also, I've investigated the mail backlog on master and found the main problem. The mail queue is currently full of email that will never be able to be delivered, all for one particular user. This mail is being removed from the queue, and the setup changed to not deliver to the problem address. Once this is finished (a few days), mail latency for mail moving through master should be greatly improved. I have now discovered that the `one user' referred to is at least two of the users of my colo system, chiark.greenend.org.uk. I am upset because I was not told of the problem and given any chance to help fix it, and because the announcement was sufficiently inaccurate that even though I read it and even investigated in my logs, I concluded - erroneously but without error on my part - that chiark wasn't involved. I would like to point out the following facts: Regarding the announcement and its lack of accuracy, and the lack of communication with the persons responsible: * At least two people with @debian.org addresses have mail forwarded to chiark: myself [EMAIL PROTECTED] and Alan Bain [EMAIL PROTECTED]. * Several other Debian developers have chiark addresses, or have mail domains hosted by chiark. * When I read the announcement I read it carefully and also searched my logs, to check whether my systems were involved. On discovering in my logs quite a few rejections of mail to _two_ users, I concluded that since the announcement talked about `one particular user' (not `one particular host'), I was not affected. * I now discover by reading master's exim4.conf that all mail to the _mail domain_ chiark.greenend.org.uk has been arranged to bounce on master. This is contrary to what is stated in the announcement, which says that the setup was changed `to not deliver to the problem _address_' (emph. mine). * No serious attempt has been made to contact me about this problem, either in my role as sysadmin for the affected host, or as one of the affected users. [EMAIL PROTECTED] would have been happy to help for example - and yes, a message to postmaster would have got through. * No clear thought was given by the system administrators as to the possible collateral damage of never attempting to deliver mail to a particular mail host. One example of such collateral damage is that several members of the Technical Committee receive mails for debian-ctte-private _via_ an alias on chiark; we set up this arrangement after the Debian systems proved unable to maintain a simple mail alias without occasionally having people fall off it ! Regarding the problem and how to solve it: * The mail backlog that `will never be able to be delivered' was (as far as I can tell) all spam that chiark has been properly rejecting. Usually, the messages have invalid envelope senders. * It is unfortunate that (a) master has such a lax spam policy and that (b) Debian developers cannot choose to make their @debian.org address unuseable other than by the Debian system administrators. This problem (lax policy at more-upstream mail host) always results in a lot of downstream rejections. * I guess that part of the problem was that the volume of mail rejections (92 rejected SMTP commands in the week ending 2005-10-23 11:42 UTC) engaged chiark's teergrube feature. This deliberately delays the SMTP responses to hosts which are generating lots of errors, because those hosts are usually spammers or zombies. * chiark has had the ability, since September 2003, to mark certain mail flows as `OK to have lots of errors' (this feature postdates the mail setups for my and Alan Bain's mail forwarding, which is why it wasn't already configured for those). That would prevent the teergrube, although it would not prevent the spam rejection of course. I have made this configuration change for my personal account and I will be forwarding a copy of this mail to Alan. * master should immediately be configured as follows: 1. The special case for chiark in exim4.conf should be removed 2. If this will make the situation untenable, mail for [EMAIL PROTECTED] should be rejected by master at SMTP-time, probably by using rootly powers to edit ~afrb2/.forward. 3. Alan should be notified of 2. and asked to sort it out. Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
On Tue, Nov 15, 2005 at 12:18:45PM +, Ian Jackson wrote: Ryan Murray writes (master's mail backlog and upgrade time): Also, I've investigated the mail backlog on master and found the main problem. The mail queue is currently full of email that will never be able to be delivered, all for one particular user. This mail is being removed from the queue, and the setup changed to not deliver to the problem address. Once this is finished (a few days), mail latency for mail moving through master should be greatly improved. I have now discovered that the `one user' referred to is at least two of the users of my colo system, chiark.greenend.org.uk. Based on specifics (well... more-specific vaguenesses) mentioned by Ryan elsewhere, I don't believe this is the case. Chiark appears to be on the wrong continent to be attached to the user in question, and reducing one to two seems like a rather incredible off-by-one error to make in this case. * I now discover by reading master's exim4.conf that all mail to the _mail domain_ chiark.greenend.org.uk has been arranged to bounce on master. This is contrary to what is stated in the announcement, which says that the setup was changed `to not deliver to the problem _address_' (emph. mine). Well, I certainly see a line in that file relating to chiark, but I have no exim-fu to understand the precise semantics -- and certainly no idea why it's there, sorry. Regarding the problem and how to solve it: * The mail backlog that `will never be able to be delivered' was (as far as I can tell) all spam that chiark has been properly rejecting. No: there is nothing proper about rejecting mail from a host that you have configured to forward mail for you. * It is unfortunate that (a) master has such a lax spam policy and that (b) Debian developers cannot choose to make their @debian.org address unuseable other than by the Debian system administrators. ... procmail? -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. [EMAIL PROTECTED] http://www.debian.org/ signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
Steve Langasek writes (Re: master's mail backlog and upgrade time): Based on specifics (well... more-specific vaguenesses) mentioned by Ryan elsewhere, I don't believe this is the case. Chiark appears to be on the wrong continent to be attached to the user in question, and reducing one to two seems like a rather incredible off-by-one error to make in this case. ... Well, I certainly see a line in that file relating to chiark, but I have no exim-fu to understand the precise semantics -- and certainly no idea why it's there, sorry. The configuration makes the mail domain disappear as far as master is concerned. The main effect is that master will never attempt to deliver mail to the mail domain chiark.greenend.org.uk. That is, any recipient of the form something@chiark.greenend.org.uk will be rejected or bounced (as the case may be) with `Unrouteable mail domain'. As a side-effect, any addresses forwarded to addresses @chiark (eg, [EMAIL PROTECTED], which I do not publish anywhere and would rather get rid of) are also undeliverable. Note, as a subtlety, that this does /not/ affect mail sent to any other domain for which chiark is the MX. If a domain was set up to be treated this way for an unrelated reasons without an announcement anywhere, surely that is even worse ! On Tue, Nov 15, 2005 at 12:18:45PM +, Ian Jackson wrote: * The mail backlog that `will never be able to be delivered' was (as far as I can tell) all spam that chiark has been properly rejecting. No: there is nothing proper about rejecting mail from a host that you have configured to forward mail for you. Nearly all of this mail flow is invalid in one way or another. The most common case is that when chiark asks the MX for the envelope sender about the envelope sender address, it turns out to be invalid or unverifiable. What is happening is that master provides a mail forwarding facility with a lax input policy, but I forward my mail to chiark which does stricter checks and has a stricter policy. This has the effect of turning the rejected messages into bounced bounces on master. This would be an unfriendly thing to do if the sysadmins on master actually cared about reliable mail delivery and took a policy of reviewing bounced bounces and dealing with their causes (as I do on chiark). As it is, all it does is cause a bit of work for computers as chiark is offered the mail, rejects it, and master then turns it into a bounced bounce which will be discarded. The problem with this using too much of master's capacity was due to the teergrube (tarpit) feature as I described in my last mail. It is of course a bad idea to treat incoming /forwarded/ junkmail flows as a reason for teergrube because it just clogs up the (friendly) forwarding host. That's why I have disabled this for my account (and would have done so straight away if asked). But, there is another important point: I don't really want a debian.org address. It's technically necessary for me to have one for (eg) cronmail from debian systems, and additionally I feel that there is an (unwritten) rule that I should have one. But it is simply untenable to suggest that I ought to accept all of this junkmail and actually read it ! Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
Steve Langasek writes (Re: master's mail backlog and upgrade time): On Tue, Nov 15, 2005 at 12:18:45PM +, Ian Jackson wrote: * It is unfortunate that (a) master has such a lax spam policy and that (b) Debian developers cannot choose to make their @debian.org address unuseable other than by the Debian system administrators. ... procmail? How would that help ? I could arrange for procmail to try to detect whether (eg) the mail had ever been through any non-debian.org host and if so construct a bounce. But this is no better and no worse than me having chiark rejecting the mails at SMTP time. It would still lead to master having to deal with lots of undeliverable bounces. What I really want is for master (et al) to say 550 at SMTP time to any incoming mail for my debian.org address, unless the source is another Debian host (as determined by IP address, not just HELO). I don't believe that there is a mechanism available for me to do that. Ian. (Sorry, I forgot to reply to this in my last mail.) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
On Nov 15, Ian Jackson [EMAIL PROTECTED] wrote: But, there is another important point: I don't really want a debian.org address. Me neither. In the past the debian-admins suggested that they would consider allowing to disable them if somebody else implemented everything needed to do it. -- ciao, Marco signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
Marco d'Itri writes (Re: master's mail backlog and upgrade time): [I don't want a debian.org address either]. In the past the debian-admins suggested that they would consider allowing to disable them if somebody else implemented everything needed to do it. Do we know what would be needed ? The biggest thing, it seems to me, would be to ensure that nothing on the debian.org systems ever generated mails to username@debian.org. This is easily done for cron with MAILTO=..., and hardly anyone uses `at' any more. But there might be other scripts which would call sendmail directly or indirectly and something appropriate would have to be put in the envelope sender of these mails. In the past, one reason for keeping the address was that that was the only way for the Debian admins to contact everyone with an account. But nowadays we can use the developer DB for that, I think ? Ian. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
On Nov 15, Ian Jackson [EMAIL PROTECTED] wrote: [I don't want a debian.org address either]. In the past the debian-admins suggested that they would consider allowing to disable them if somebody else implemented everything needed to do it. Do we know what would be needed ? An updated exim configuration which can look at the LDAP database, I suppose. I do not know the details of how debian.org mail is handled. (And anyway it would not be resolutive, I get most of my debian spam from the 23 @packages.debian.org addresses I receive...) -- ciao, Marco signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
Hi Ryan, On Thu, Nov 03, 2005 at 12:38:43PM -0800, Ryan Murray wrote: Also, I've investigated the mail backlog on master and found the main problem. The mail queue is currently full of email that will never be able to be delivered, all for one particular user. Why would that be? Could you be a little bit more specific here? This mail is being removed from the queue, and the setup changed to not deliver to the problem address. Once this is finished (a few days), mail latency for mail moving through master should be greatly improved. Well, the following mail still spent six days between the bug tracking system and master. Received: from master.debian.org (master.debian.org [146.82.138.7]) by dd1234.kasserver.com (Postfix) with ESMTP id EEE0A17785E for [EMAIL PROTECTED]; Sun, 6 Nov 2005 05:40:47 +0100 (CET) Received: from qa by master.debian.org with local (Exim 3.35 1 (Debian)) id 1EYcKX-0006Z7-00; Sat, 05 Nov 2005 22:40:33 -0600 Received: from gluck.debian.org [192.25.206.10] by master.debian.org with esmtp (Exim 3.35 1 (Debian)) id 1EYcKW-0006Ya-00; Sat, 05 Nov 2005 22:40:32 -0600 Received: from spohr.debian.org [140.211.166.43] (mail) by gluck.debian.org with esmtp (Exim 3.35 1 (Debian)) id 1EWaUA-0004Jr-00; Mon, 31 Oct 2005 07:18:06 -0700 Received: from debbugs by spohr.debian.org with local (Exim 3.36 1 (Debian)) id 1EWaU8-0005n5-00; Mon, 31 Oct 2005 06:18:04 -0800 Message-Id: [EMAIL PROTECTED] All the best, Jochen -- http://seehuhn.de/ signature.asc Description: Digital signature
Re: master's mail backlog and upgrade time
On Thu, 03 Nov 2005, Ryan Murray wrote: Also, I've investigated the mail backlog on master and found the main problem. The mail queue is currently full of email that will never be able to be delivered, all for one particular user. This mail is being Can you give us more data on this? -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: master's mail backlog and upgrade time
On Thu, 2005-11-03 at 12:38 -0800, Ryan Murray wrote: Also, I've investigated the mail backlog on master and found the main problem. The mail queue is currently full of email that will never be able to be delivered, all for one particular user. This mail is being removed from the queue, and the setup changed to not deliver to the problem address. Once this is finished (a few days), mail latency for mail moving through master should be greatly improved. Thanks for sorting it out, Ryan. Drew -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
