Re: Kernel Oops
mouss put forth on 3/6/2011 7:03 PM:
/^.*foo/
means it starts with something followed by foo. and this is the same
thing as it contains foo, which is represented by
/foo/
I was taught to always start my expressions with /^ and end them with
$/. Why did Steven teach me to do this if it's not necessary? Steven
being the author of the Enemies List: http://enemieslist.com/ which
contains over 65,000 regexes matching FQrDNS patterns.
well, you know I know these:) we all got spam from these...
As with most/all dynamic ranges.
1) first use IP ranges.
2) then domains (hash/cdb)
for example:
.alshamil.net.ae REJECT blah blah
because there is no point to try to match something like
auh-b113917.alshamil.net.ae
3) then use regular expressions, but only when IPs and domains aren't
the way to go.
Well, you know I know these mouss. :) Have ever been locked in a
certain train of thought and simply forgot to consider something
related, later putting hand to forehead and saying Duh!. My mindset
was focused on showing how a single PCRE can block the same number of
hosts as using IP addresses in a CIDR or hash table. I just didn't
consider the domain blocking aspect of hash tables at the time. That's
the Duh!. I've been blocking domains with my hash table for something
like 6 years now... I think some folks call this a brain fart. ;)
no. IPs and domains are different things.
cidr is about IPs. hash/cdb/pcre is about names. these are different
things and you know that. use each as appropriate.
Of course. But IPs are valid in a hash table. You can even list them
by the equivalent of a /24, /16, and /8 if you like, simply by omitting
the last 1, 2, or 3 octets of the dotted quad. Just as I brain farted
WRT using domains in a hash table, it appears you have done the same WRT
to using IP addresses in a hash table. :)
I agree it makes more sense to block domains with hash/cdb and IPs with
CIDR. I've been doing exactly that for 5 of the 6 years I've been
running Postfix. The first year (maybe less) I blocked IPs with a hash
table, until I joined this list and learned about CIDR tables. I'm
guessing most other new Postfix OPs go through the same
progression--most beginners docs returned via Google teach the hash
table and nothing else.
if the ISP makes it too much, then you should reduce it:
.embarqhsd.netREJECT blah blah
Yeah, but then you end up potentially blocking large numbers of ham
servers in SOHO land, in this case *.sta.embarqhsd.net. Even in 2011
there are still hundreds of thousands or more SOHO MTAs on static IP
aDSL and cable circuits with generic rDNS. I should know as I'm one of
them. (Please let's not allow this to turn into yet another flame war
WRT generic rDNS, real OPs rent a VPS/colo, yada yada--I'm not directing
this at you mouss but to those predisposed to flog this dead, stripped
to the bone, horse carcass).
a better example would be
/(\W\d+){4}\..*\.embarqhsd\.net$/ REJECT ...
Better in what way?
in the sense that this can't be represented using hash or the like.
Ok. So you're not showing this PCRE above because it better matches the
target rDNS string, or that the engine executes it faster or something,
etc. You're simply saying don't use a PCRE for something you can match
using a simpler table, such as hash/cdb. Correct?
--
Stan
Re: Looking for instructions on how to configure home server as a restricted relay host
Reid Thompson put forth on 3/6/2011 9:07 PM: What I would like to do: Configure my home postfix server (ubuntu) to: send email from local user accounts accept external (through my cable modem) smtp requests/relay mail for only authorized senders I.E. when I'm using a public internet connection, i'd like to have my smtp requests go through my home server Could someone point me to a website describing how to configure this? http://www.postfix.org/SOHO_README.html -- Stan
Re: Dovecot, Postfix and Dovecot LDA (LMTP) delivery
I am running a Debian Lenny machine with Postfix 2.5.5 and Dovecot 2.0.8.
Up until now I ran Postfix with the Procmail delivery agent succesfully.
The machine has only local users; I am not using virtual mailboxes. Due to
obvious reasons I want to switch to Dovecot LDA delivery, through LMTP.
I spend quite some time finding 'the' configuration for both Postfix and
Dovecot, but without success. Find my current configuration below. Postfix
receives the e-mail, but delivery through LMTP fails, resulting in Postfix
sending an NDR.
I just cannot figure out what's wrong and I am not sure whether it's a
Postfix or Dovecot problem either. Some log output from Postfix:
I finally figured out what's wrong. It appears that Dovecot in fact is
checking the existance of user 'r...@hostname.domain.tld' in one of the
configured user databases. Obviously, local users/usernames do not have the
local domain added.
Adding
userdb {
driver = passwd-file
args = username_format=%n /etc/passwd
}
to the 'protocol lmtp { }' block solves the issue.
For archival purposes only:
On the Dovecot list I was told using the above method is not the best idea,
since there are no guarantees it will always work without failures.
The best way to solve the problem is to configure
auth_username_format = %n
in conf.d/10-auth.conf
-Remy
Re: Configuration of postfix 2.8.1 + ezmlm 1.2.17
On Sun, 6 Mar 2011 18:46:44 -0500 (EST), Wietse Venema wie...@porcupine.org wrote: In order to have postfix 2.8.1 feeding email to a ezmlm 1.2.17 If you follow the mlmmj website's instructions, then it should work. Do you mean the README.Postfix at http://mlmmj.org/archive/mlmmj/att-0511/README.postfix ? I prefer not to review alternative variations. I can understand that. But in this case, as this is rather generic regarding MLM proper interfacing with Postfix, it sure would be welcome to have your view about the proper way to setup a generic interface between Postfix and a generic MLM. By the way, the mlmmj setup can now be simplified, and no longer needs the kludge with the mlmmj/pipe transport. I know that it also consumes time but... could you give a tiny example/usage case? postfix-2.9-20110228 fixes a problem where the local delivery agent ignored the ownership of regexp-based alias tables. Will this bug fix be backported into 2.8.x ? Why was this fixed 20110228? Because I recently stumbled upon this problem when I visited the mlmmj/postfix webpage. That page is marked Nov 12th 2005 but in fact, as it is clear from the change log, it just appeared in the latest (1.2.17.1, Nov 2010) release. M.
Re: Kernel Oops
On 2011-03-07 Stan Hoeppner wrote: mouss put forth on 3/6/2011 7:03 PM: /^.*foo/ means it starts with something followed by foo. and this is the same thing as it contains foo, which is represented by /foo/ I was taught to always start my expressions with /^ and end them with $/. Why did Steven teach me to do this if it's not necessary? I wouldn't know what his rationale was, but Noel and mouss are certainly right. Anchoring something between wildcard matches is utterly pointless. As mouss explained above, /^.*foo/, /.*foo/ and /foo/ produce the same results. That is, unless your regexp processor implicitly anchors an expression at the beginning of the string, in which case you'd need the leading .*, but still won't need to explicitly anchor it with a ^. Regards Ansgar Wiechers -- Abstractions save us time working, but they don't save us time learning. --Joel Spolsky
Re: submission port : Client host rejected: Access denied
On 6 mrt 2011, at 22:34, Noel Jones wrote: On 3/6/2011 9:08 AM, DTNX/NGMX Postmaster wrote: I suspect that if you were to increase logging detail, you'd find that 'permit_sasl_authenticated' evaluates to zero during the client restrictions stage because of a delay in getting back an answer from whatever SASL backend you have in use. Postfix evaluates the rest of the client restrictions, and denies you access. No. The SASL authentication happens after CONNECT and HELO, before MAIL FROM. With smtpd_delay_reject = no, and smtpd_client_restrictions = permit_sasl_authenticated, reject you're checking for sasl authentication before the authentication ever has a chance to take place. This has nothing to do with what you're using for a sasl backend, because the backend is never consulted. Just another good reason to not muck with the defaults. Hmm, I must be remembering it wrong then, because that makes perfect sense. Or I interpreted the logging data incorrectly, which is not impossible either. Anyway, thanks for the correction. Cya, Jona
Re: Kernel Oops
On 3/7/2011 4:47 AM, Stan Hoeppner wrote: I was taught to always start my expressions with /^ and end them with $/. Why did Steven teach me to do this if it's not necessary? That's good advice when you're actually matching something. The special case of .* means, as you know, anything or nothing. There's never a case where it's necessary to explicitly match a leading or trailing anything or nothing. Consider: /^.*foo$/ match the string beginning with anything or nothing, ending with foo. can always be simplified to: /foo$/ match the string ending with foo. This works the same without the ending $ anchor (contains foo, rather than ends with foo), but helps the illustration. (In the other special case where you're using $1, $2, etc. substitution in the result, you might need some form of /^(.*foo)$/ to fill the substitution buffer, but that's about substitution, not about matching.) -- Noel Jones
Re: Re : Re : Re : Re : Re : Re : Re : slow transport, master.cf and maxproc value
myrdhin bzh put forth on 3/7/2011 1:53 AM: No, it's not SPAM. Ok, agreed, not spam. In fact, zeDomain.tld is a french know domain : wanadoo.fr (and orange.fr). :( The same 10 servers handle mail for both wanadoo.fr and orange.fr. All Postfix controls relevant to your issue available in 2.1.5 are based on connection concurrency, not rate limiting. The connection concurrency is on a *per domain* basis. Orange is the largest ISP in France, yes? How many other recipient domains you send mail to are also hosted by this same Orange 10 server mail farm? If your answer is a lot, you could completely disable concurrency and still possibly trip their 3 connections per server limit, simply due to the number of recipient domains for which you have mail queued. They didn't list them in the same order, but if you sort them you'll see it's the same 10 servers. smtp.wanadoo.fr.600 IN A 80.12.242.62 smtp.wanadoo.fr.600 IN A 80.12.242.148 smtp.wanadoo.fr.600 IN A 193.252.22.65 smtp.wanadoo.fr.600 IN A 193.252.23.67 smtp.wanadoo.fr.600 IN A 80.12.242.9 smtp.wanadoo.fr.600 IN A 80.12.242.53 smtp.wanadoo.fr.600 IN A 80.12.242.142 smtp.wanadoo.fr.600 IN A 80.12.242.82 smtp.wanadoo.fr.600 IN A 193.252.22.92 smtp.wanadoo.fr.600 IN A 80.12.242.15 smtp-in.orange.fr. 600 IN A 80.12.242.148 smtp-in.orange.fr. 600 IN A 80.12.242.53 smtp-in.orange.fr. 600 IN A 80.12.242.9 smtp-in.orange.fr. 600 IN A 193.252.23.67 smtp-in.orange.fr. 600 IN A 193.252.22.65 smtp-in.orange.fr. 600 IN A 80.12.242.142 smtp-in.orange.fr. 600 IN A 80.12.242.62 smtp-in.orange.fr. 600 IN A 80.12.242.82 smtp-in.orange.fr. 600 IN A 193.252.22.92 smtp-in.orange.fr. 600 IN A 80.12.242.15 I would suggest you grep your mail log for all outbound smtp connections to these 10 IP addresses, and document all the recipient domains. Then add all these domains to your slow transport map. If you still get those 421 errors, completely disable concurrency. If you still get the errors, you probably can't fix the problem until you upgrade to 2.5+ and have access to rate delay controls. Hope this helps. -- Stan
Re: Re : Re : Re : Re : Re : Re : Re : slow transport, master.cf and maxproc value
On 3/7/2011 7:21 AM, Stan Hoeppner wrote: myrdhin bzh put forth on 3/7/2011 1:53 AM: No, it's not SPAM. Ok, agreed, not spam. In fact, zeDomain.tld is a french know domain : wanadoo.fr (and orange.fr). :( The same 10 servers handle mail for both wanadoo.fr and orange.fr. All Postfix controls relevant to your issue available in 2.1.5 are based on connection concurrency, not rate limiting. The connection concurrency is on a *per domain* basis. Orange is the largest ISP in France, yes? How many other recipient domains you send mail to are also hosted by this same Orange 10 server mail farm? If your answer is a lot, you could completely disable concurrency and still possibly trip their 3 connections per server limit, simply due to the number of recipient domains for which you have mail queued. They didn't list them in the same order, but if you sort them you'll see it's the same 10 servers. smtp.wanadoo.fr.600 IN A 80.12.242.62 smtp.wanadoo.fr.600 IN A 80.12.242.148 smtp.wanadoo.fr.600 IN A 193.252.22.65 smtp.wanadoo.fr.600 IN A 193.252.23.67 smtp.wanadoo.fr.600 IN A 80.12.242.9 smtp.wanadoo.fr.600 IN A 80.12.242.53 smtp.wanadoo.fr.600 IN A 80.12.242.142 smtp.wanadoo.fr.600 IN A 80.12.242.82 smtp.wanadoo.fr.600 IN A 193.252.22.92 smtp.wanadoo.fr.600 IN A 80.12.242.15 smtp-in.orange.fr. 600 IN A 80.12.242.148 smtp-in.orange.fr. 600 IN A 80.12.242.53 smtp-in.orange.fr. 600 IN A 80.12.242.9 smtp-in.orange.fr. 600 IN A 193.252.23.67 smtp-in.orange.fr. 600 IN A 193.252.22.65 smtp-in.orange.fr. 600 IN A 80.12.242.142 smtp-in.orange.fr. 600 IN A 80.12.242.62 smtp-in.orange.fr. 600 IN A 80.12.242.82 smtp-in.orange.fr. 600 IN A 193.252.22.92 smtp-in.orange.fr. 600 IN A 80.12.242.15 I would suggest you grep your mail log for all outbound smtp connections to these 10 IP addresses, and document all the recipient domains. Then add all these domains to your slow transport map. If you still get those 421 errors, completely disable concurrency. Nice explanation and reasonable advice. If you still get the errors, you probably can't fix the problem until you upgrade to 2.5+ and have access to rate delay controls. Unfortunately, the newer postfix rate delay controls are still based on the recipient domain, not the MX destination. Maybe some clever firewall rules could help. Hmm, I'll wonder out loud if a check_recipient_mx_access table that returns FILTER slow: would help. It would affect all recipients of a multi-recipient message, but maybe that would be acceptable. -- Noel Jones
Re: Kernel Oops
Noel Jones put forth on 3/7/2011 7:00 AM:
On 3/7/2011 4:47 AM, Stan Hoeppner wrote:
I was taught to always start my expressions with /^ and end them with
$/. Why did Steven teach me to do this if it's not necessary?
That's good advice when you're actually matching something.
Ok, so if I'm doing what I've heard called a fully qualified regular
expression, WRT FQrDNS matching, should I use the anchors or not?
postmap -q says these all work (the actuals with action and text that is).
/^(\d{1,3}-){3}\d{1,3}\.dynamic\.chello\.sk$/
/^(\d{1,3}\.){4}dsl\.dyn\.forthnet\.gr$/
/^(\d{1,3}-){4}adsl-dyn\.4u\.com\.gh$/
/^[\d\w]{8}\.[\w]{2}-[\d]-[\d\w]{2}\.dynamic\.ziggo\.nl$/
/^(\d{1,3}\.){4}dynamic\.snap\.net\.nz$/
/^pppoe-dyn(-\d{1,3}){4}\.kosnet\.ru$/
The special case of .* means, as you know, anything or nothing.
There's never a case where it's necessary to explicitly match a leading
or trailing anything or nothing.
What of the case where you want to match something in the middle of the
input string, with extra junk on both ends?
Consider:
/^.*foo$/
match the string beginning with anything or nothing, ending with foo.
can always be simplified to:
/foo$/
match the string ending with foo.
This works the same without the ending $ anchor (contains foo, rather
than ends with foo), but helps the illustration.
So, in my examples above, given we're matching rDNS patterns, are the
anchors necessary, or helpful? If not using them means contains, then
they should still match. What advantage is there to using the anchors
when matching rDNS patterns? Any?
(In the other special case where you're using $1, $2, etc. substitution
in the result, you might need some form of /^(.*foo)$/ to fill the
substitution buffer, but that's about substitution, not about matching.)
Thank you for the continuing PCRE education Noel, and Ansgar. :)
--
Stan
Re: Configuration of postfix 2.8.1 + ezmlm 1.2.17
Mark Alan: But in this case, as this is rather generic regarding MLM proper interfacing with Postfix, it sure would be welcome to have your view about the proper way to setup a generic interface between Postfix and a generic MLM. Sorry, I don't have time to review all the instructions on the web that describe how to plug some program into Postfix. That's what I have to leave to the people who use that program with Postfix. By the way, the mlmmj setup can now be simplified, and no longer needs the kludge with the mlmmj/pipe transport. I know that it also consumes time but... could you give a tiny example/usage case? Consider this your opportunity to become a hero, with help from the mailing list :-) The basic idea is that with a local aliases file, file ownership determines the execution privileges for |command and /file/name destinations, and the envelope sender address for non-delivery notifications. In other words, alias ownership of regexp/pcre files now works as documented. Just give the file the right owner, and |command will execute as the owner of the aliases file, instead of nobody. postfix-2.9-20110228 fixes a problem where the local delivery agent ignored the ownership of regexp-based alias tables. Will this bug fix be backported into 2.8.x ? Never. This was an invasive code change that affected 25 different source code files. I will be proofreading and testing that code several times over the course of this year before it becomes part of the next stable release. Wietse
Re: Kernel Oops
On 3/7/2011 8:13 AM, Stan Hoeppner wrote:
Noel Jones put forth on 3/7/2011 7:00 AM:
On 3/7/2011 4:47 AM, Stan Hoeppner wrote:
I was taught to always start my expressions with /^ and end them with
$/. Why did Steven teach me to do this if it's not necessary?
That's good advice when you're actually matching something.
Ok, so if I'm doing what I've heard called a fully qualified regular
expression, WRT FQrDNS matching, should I use the anchors or not?
postmap -q says these all work (the actuals with action and text that is).
/^(\d{1,3}-){3}\d{1,3}\.dynamic\.chello\.sk$/
/^(\d{1,3}\.){4}dsl\.dyn\.forthnet\.gr$/
/^(\d{1,3}-){4}adsl-dyn\.4u\.com\.gh$/
/^[\d\w]{8}\.[\w]{2}-[\d]-[\d\w]{2}\.dynamic\.ziggo\.nl$/
/^(\d{1,3}\.){4}dynamic\.snap\.net\.nz$/
/^pppoe-dyn(-\d{1,3}){4}\.kosnet\.ru$/
In these examples, you're explicitly matching something at the
start and/or end of the string. Using the anchors is correct
and recommended.
The special case of .* means, as you know, anything or nothing.
There's never a case where it's necessary to explicitly match a leading
or trailing anything or nothing.
What of the case where you want to match something in the middle of the
input string, with extra junk on both ends?
If you're looking for a string that contains foo anywhere, simply
/foo/
with no anchors.
Consider:
/^.*foo$/
match the string beginning with anything or nothing, ending with foo.
can always be simplified to:
/foo$/
match the string ending with foo.
This works the same without the ending $ anchor (contains foo, rather
than ends with foo), but helps the illustration.
So, in my examples above, given we're matching rDNS patterns, are the
anchors necessary, or helpful? If not using them means contains, then
they should still match. What advantage is there to using the anchors
when matching rDNS patterns? Any?
You use anchors to reduce the chance of a false positive. A
side benefit is improved performance.
Any pattern that matches with the anchors will still match
without the anchors, but may match additional input that you
don't intend to match. In the case of the rDNS patterns, a FP
is unlikely (but possible, more so with the shorter patterns).
In other cases, such as matching a sort bare domain name, a FP
may be very likely without anchors.
best practice is to use the anchors when you can, ie. what
you're matching will always be at the beginning and/or end of
the input string. Never use ^.* or .*$.
-- Noel Jones
Re: Configuration of postfix 2.8.1 + ezmlm 1.2.17
On Mon, 7 Mar 2011 09:43:40 -0500 (EST), Wietse Venema wie...@porcupine.org wrote: The basic idea is that with a local aliases file, file ownership determines the execution privileges for |command and /file/name destinations, and the envelope sender address for non-delivery notifications. Meaning that (keeping with the example lis...@example.org mailing list) the following would simply work as expected: /etc/postfix/virtual-alias-maps lis...@example.org list01@localhost /etc/aliases list01: |/usr/bin/mlmmj-recieve -L /var/spool/mlmmj/list01/ In other words, alias ownership of regexp/pcre files now works as documented. If not for anything else, at least this was good to fix that bug. ... and, by the way, the subject of this thread should have been Configuration of postfix 2.8.1 + mlmmj 1.2.17, not ezmlm... but it seems that old habits don't die easily. Thank you Wietse and keep up this great work M.
Re: Mails in database.
I've followed the discussion in the thread. Is it possible to for example make configuration with: - two SMTP(Postfix)/POPIMAP(maildb) servers in front; - two db servers(PostgreSQL) working in active-active (both readwrite) configuration behind? I would like to have configuration in which failure of one of front or/and one of database nodes would not make the system unusable. Also I would like to use both database servers (readwrite) during normal operation to share load beetwen them. Is it possible? I'm not a database professional so please excuse me if the question is fairly easy :) You have written that one can use MySQL and read-only slaves. These slave servers can be used for handling read only user queries (POPIMAP). I have a second question: POPIMAP do not only make read operations, these protocols also support write operations (make a directory, remove a message, ... - as I think). So what is the purpose of read-only database host during normal operation (when active/master node is working properly)? As I see it now it can surely be used as a spare or for creating reports, are there more usage possibilities? Regards, Rafal. 2011/3/5 Reindl Harald h.rei...@thelounge.net Am 05.03.2011 14:13, schrieb Stan Hoeppner: Reindl Harald put forth on 3/5/2011 6:02 AM: Thanks for the info. Need one clarification: Can you cluster the dbmail IMAP daemon on multiple external hosts to support thousands of concurrent IMAP users, without the locking contention of NFS or cluster filesystems, thus achieving lower latency and greater throughput? yes you can because you have only to install dbmail/postfix on all of them with access to the same database, for postfix you can use replication-slaves too as fallback I asked about multiple IMAP servers and you answered with Postifx, which is an SMTP server, not an IMAP server. it seems you did not read i answered with dbmail/postfix dbmail = imap/pop3/lmtp/sieve postfix belongs to my answer because i would use one big db-server and on the mail-hosts postfix/dbmail-imapd/dbmail-pop3/dbmail-lmtpd sahring the same database and the point is that you can use all this hosts as mx and imap-host because they have the same data using replication slaves for postfix is nice beacuse you can spread the read-only-load away from the main-db-server In the Dovecot world, for a high availability and high concurrent user load cluster, one may setup say, 12 low end 1U rack servers with 4GB RAM and dual GbE ports each, to handle about 5,000 concurrent IMAP users (~400 users each), with all 12 servers accessing the same high performance NFS mailstore. and in dbmail you use a hig-performance db-server Dovecot can't use dbmail for message storage or indexes. dovecot is in context dbmail obsolete but we use it as imap/pop3-proxy and for authentication in front of dbmail because dovecot supports more auth-mechs My question is how would one build such an IMAP cluster with dbmail? I would assume a dbmail IMAP server component would need to replace Dovecot in this setup. Is such a thing possible? yes since dbmail is imap/pop3/lmtp/sieved Apologies to all as this thread has wandered into OT territory. This is my last post on the subject. I just wanted to understand if/how dbmail can scale with both Postfix delivery and IMAP retrieval in a clustered setup. no problem, you did not realize that dbmail/mysqld is the whole mail-system only needing an MTA like postfix or exim
Re: Mails in database.
Am 07.03.2011 17:14, schrieb Rafał Radecki: I've followed the discussion in the thread. Is it possible to for example make configuration with: - two SMTP(Postfix)/POPIMAP(maildb) servers in front; - two db servers(PostgreSQL) working in active-active (both readwrite) configuration behind? this is a question for postgresql-mailing-list dbmail does not interest how are your db-servers are consistent you install dbmail and say what db-server it should use, that was it how you get a db-cluster with your needs is another game I would like to have configuration in which failure of one of front or/and one of database nodes would not make the system unusable. Also I would like to use both database servers (readwrite) during normal operation to share load beetwen them. Is it possible? I'm not a database professional so please excuse me if the question is fairly easy :) as said, i am using mysql with a read-only salve (my.cnf-param) You have written that one can use MySQL and read-only slaves. yes, becasue they are easy These slave servers can be used for handling read only user queries (POPIMAP). no i said nothing about use them for imap/pop3 dbmail does not make a difference read/write I have a second question: POPIMAP do not only make read operations, these protocols also support write operations (make a directory, remove a message, ... - as I think). yes and that is why i said postfix not dbmail So what is the purpose of read-only database host during normal operation (when active/master node is working properly)? As I see it now it can surely be used as a spare or for creating reports, are there more usage possibilities? postfix is enough for receive mails and queue them until the other parts of the mail-system are back and after that they are delivered Regards, Rafal. 2011/3/5 Reindl Harald h.rei...@thelounge.net mailto:h.rei...@thelounge.net Am 05.03.2011 14:13, schrieb Stan Hoeppner: Reindl Harald put forth on 3/5/2011 6:02 AM: Thanks for the info. Need one clarification: Can you cluster the dbmail IMAP daemon on multiple external hosts to support thousands of concurrent IMAP users, without the locking contention of NFS or cluster filesystems, thus achieving lower latency and greater throughput? yes you can because you have only to install dbmail/postfix on all of them with access to the same database, for postfix you can use replication-slaves too as fallback I asked about multiple IMAP servers and you answered with Postifx, which is an SMTP server, not an IMAP server. it seems you did not read i answered with dbmail/postfix dbmail = imap/pop3/lmtp/sieve postfix belongs to my answer because i would use one big db-server and on the mail-hosts postfix/dbmail-imapd/dbmail-pop3/dbmail-lmtpd sahring the same database and the point is that you can use all this hosts as mx and imap-host because they have the same data using replication slaves for postfix is nice beacuse you can spread the read-only-load away from the main-db-server In the Dovecot world, for a high availability and high concurrent user load cluster, one may setup say, 12 low end 1U rack servers with 4GB RAM and dual GbE ports each, to handle about 5,000 concurrent IMAP users (~400 users each), with all 12 servers accessing the same high performance NFS mailstore. and in dbmail you use a hig-performance db-server Dovecot can't use dbmail for message storage or indexes. dovecot is in context dbmail obsolete but we use it as imap/pop3-proxy and for authentication in front of dbmail because dovecot supports more auth-mechs My question is how would one build such an IMAP cluster with dbmail? I would assume a dbmail IMAP server component would need to replace Dovecot in this setup. Is such a thing possible? yes since dbmail is imap/pop3/lmtp/sieved Apologies to all as this thread has wandered into OT territory. This is my last post on the subject. I just wanted to understand if/how dbmail can scale with both Postfix delivery and IMAP retrieval in a clustered setup. no problem, you did not realize that dbmail/mysqld is the whole mail-system only needing an MTA like postfix or exim -- Mit besten Grüßen, Reindl Harald the lounge interactive design GmbH A-1060 Vienna, Hofmühlgasse 17 CTO / software-development / cms-solutions p: +43 (1) 595 3999 33, m: +43 (676) 40 221 40 icq: 154546673, http://www.thelounge.net/ signature.asc Description: OpenPGP digital signature
Re: posfix rejected from google server
On Mon, Mar 07, 2011 at 09:01:21AM +0900, Peter Evans wrote: At the same time, please go here and request removal from the PBL. http://www.spamhaus.org/pbl/query/PBL043205 PBL removal will not be done for a host with generic rDNS. I am in a PBL block too, but my host has custom reverse DNS. Spamhaus does say that this is a requirement for removal from the PBL. Conversely when rDNS fails or changes, PBL exemption can be revoked; mine was, once. PS to the OP: The shameless self-promotion I asked you to tolerate upthread (grr, what an annoying Webmail client yours is which breaks email threading!) was my own. I was promoting the use of a different mail list where this discussion would be on-topic. Specifically this is to say: it is OFF topic here. Therefore this will be my last post on the matter. You have received and rejected much good advice here. -- Offlist mail to this address is discarded unless /dev/rob0 or not-spam is in Subject: header
Re: Mails not received in postfix (sending out is fine )
Am 07.03.2011 17:23, schrieb sunhux G: Firstly, can someone point me to a link on setting up just a plain Unix mailbox (no LDAP, no fanciful stuff). I know how to use useradd to create a Unix account nothing more about getting it to be able to receive external emails. Is saslauth needed for postfix to be able to receive mails? no, it is only needed to prevent beeing a spam-relay to only allow relay for authenticated users how should any server make auth while delivering to you? :-) I've deinstalled sendmail from my Linux postfix box reinstalled postfix. fine Doing a find from root, there's only one copy of main.cf one copy of master.cf in the whole system. fine On the postfix server, did a test: # telnet localhost 25 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. 220 mypostfixhostname.mypostfix_domain.com ESMTP POSTFIX HELO smtp.mypostfixdomain.com 250 mypostfixhostname.mypostfixdomain.com MAIL FROM:sender_id@sending_domain.com.sg 250 2.1.0 Ok RCPT TO:myunixid_inpostfix@mypostfix_domain.com 250 2.1.5 Ok DATA 354 End data with CRLF.CRLF testg 1 testg 2 . 250 2.0.0 Ok: queued as B7613200034== can't locate mail with this queue id QUIT 221 2.0.0 Bye Connection closed by foreign host. look at the log you posted to=r...@mypostfixhostname.mypostfixdomain.com, orig_to=myunixid_inpostfix@mypostfix_domain.com, relay=local, delay=32, delays=32/0.01/0/0.05, dsn=5.4.6, status=bounced (alias database loop for root) [root@mypostfix_hostname postfix]# mailq | grep B7613200034 [root@mypostfix_hostname postfix]# tail -22 /var/log/maillog Mar 7 12:11:09 mypostfix_hostname postfix/postfix-script[9477]: stopping the Postfix mail system Mar 7 12:11:09 mypostfix_hostname postfix/master[9431]: terminating on signal 15 Mar 7 12:11:10 mypostfix_hostname postfix/postfix-script[9528]: starting the Postfix mail system Mar 7 12:11:10 mypostfix_hostname postfix/master[9529]: daemon started -- version 2.5.6, configuration /etc/postfix Mar 7 12:11:13 mypostfix_hostname postfix/postfix-script[9536]: refreshing the Postfix mail system Mar 7 12:11:13 mypostfix_hostname postfix/master[9529]: reload configuration /etc/postfix Mar 7 12:12:25 mypostfix_hostname postfix/smtpd[9575]: warning: smtpd_sasl_auth_enable is true, but SASL support is not compiled in Mar 7 12:12:25 mypostfix_hostname postfix/smtpd[9575]: connect from mypostfix_hostname[127.0.0.1] Mar 7 12:13:41 mypostfix_hostname postfix/smtpd[9575]: disconnect from mypostfix_hostname[127.0.0.1] Mar 7 12:13:42 mypostfix_hostname postfix/smtpd[9575]: connect from mypostfix_hostname[127.0.0.1] Mar 7 12:14:14 mypostfix_hostname postfix/smtpd[9575]: B7613200034: client=mypostfix_hostname[127.0.0.1] Mar 7 12:14:28 mypostfix_hostname postfix/cleanup[9594]: B7613200034: message-id=20110307041414.b7613200...@mypostfixhostname.mypostfixdomain.com Mar 7 12:14:28 mypostfix_hostname postfix/qmgr[9595]: B7613200034: from=sender_id@sending_domain.com.sg, size=398, nrcpt=1 (queue active) Mar 7 12:14:28 mypostfix_hostname postfix/local[9596]: warning: alias database loop for root Mar 7 12:14:28 mypostfix_hostname postfix/local[9596]: B7613200034: to=r...@mypostfixhostname.mypostfixdomain.com, orig_to=myunixid_inpostfix@mypostfix_domain.com, relay=local, delay=32, delays=32/0.01/0/0.05, dsn=5.4.6, status=bounced (alias database loop for root) Mar 7 12:14:28 mypostfix_hostname postfix/cleanup[9594]: 60A3B20005C: message-id=20110307041428.60a3b200...@mypostfixhostname.mypostfixdomain.com Mar 7 12:14:28 mypostfix_hostname postfix/qmgr[9595]: 60A3B20005C: from=, size=2355, nrcpt=1 (queue active) Mar 7 12:14:28 mypostfix_hostname postfix/bounce[9598]: B7613200034: sender non-delivery notification: 60A3B20005C Mar 7 12:14:28 mypostfix_hostname postfix/qmgr[9595]: B7613200034: removed Mar 7 12:14:28 mypostfix_hostname postfix/smtp[9599]: 60A3B20005C: to=sender_id@sending_domain.com.sg, relay=gate1.ncs.com.sg[203.126.130.157]:25, delay=0.35, delays=0.05/0.01/0.25/0.05, dsn=2.0.0, status=sent (250 ok: Message 76376872 accepted) Mar 7 12:14:28 mypostfix_hostname postfix/qmgr[9595]: 60A3B20005C: removed Mar 7 12:14:31 mypostfix_hostname postfix/smtpd[9575]: disconnect from mypostfix_hostname[127.0.0.1] == # postconf -n alias_database = hash:/etc/postfix/aliases alias_maps = hash:/etc/postfix/aliases bounce_size_limit = 65536 command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix data_directory = /var/lib/postfix debug_peer_level = 2 debug_peer_list = mypostfix_domain.com default_privs = nobody default_transport = smtp header_size_limit = 32768 html_directory = /usr/share/doc/postfix-2.5.6-documentation/html inet_interfaces = all local_recipient_maps = mail_owner = postfix mail_spool_directory = /pop3/spool/mail
Re: Pcre header checks
On Mon, Mar 07, 2011 at 10:39:29AM +1100, Erik de Castro Lopo wrote: I'm running postfix version 2.8.1 from Debian. I've got basic pcre header checks working as they should. I'm trying to reject mail that has a DKIM signature that says its from att.net but a Message-ID ending in 'yahoo.com'. I've got this: if /^DKIM-Signature: .*; d\=att.net;/i /^Message-ID: .*\.yahoo\.com/i REJECT endif but it doesn't seem to work. Clues? You are lucky this does not work. Much of the att.net mail infrastructure is operated by Yahoo. DKIM signatures are also added in messages handled by lists, ... What you are attemtping to do is a bad idea based on a deep misconception of the role of DKIM in email processing. -- Viktor.
Re: Re : Re : Re : Re : Re : Re : Re : slow transport, master.cf and maxproc value
Noel Jones put forth on 3/7/2011 7:37 AM: On 3/7/2011 7:21 AM, Stan Hoeppner wrote: I would suggest you grep your mail log for all outbound smtp connections to these 10 IP addresses, and document all the recipient domains. Then add all these domains to your slow transport map. If you still get those 421 errors, completely disable concurrency. Nice explanation and reasonable advice. Thanks Noel. If you still get the errors, you probably can't fix the problem until you upgrade to 2.5+ and have access to rate delay controls. Unfortunately, the newer postfix rate delay controls are still based on the recipient domain, not the MX destination. Yeah. He can absolutely fix his current problem with it. But if he has to configure huge delays to accomplish it, this obviously creates yet more problems: a backed up queue and mail not delivered in a timely manner. Maybe some clever firewall rules could help. I recall Wietse talking about implementing something a while back specifically for this case. IIRC it was something like doing an smtp_bind_address type thing with a bunch smtp clients, each bound to a different virtual IP all on a single NIC, and sending out all the IPs round robin fashion to prevent bad sender IP reputation. My memory is fuzzy on this. In lieu of this, the best solution I can think of is going multi instance, say 6 extra instances, 7 total. The following is not a complete step by step how-to but an overview of the basic steps required. On the primary instance, remove all current restrictions you've put in place WRT this Orange farm problem. Bind 6 new additional IP addresses as virtual NICs. On the primary instance do smtp_bind_address to the current existing IP address, and the same for inet_interfaces. Do this for each of the 6 new instances, so all 7 Postfix instances only use one IP address each. On each of the 6 new instances, leave smtp_connection_cache_on_demand enabled, and set smtp_destination_concurrency_limit to 18. This will allow you to have 18 concurrent smtp connections for each of 10 (primary Postfix instance) active queue destination domains hosted by the Orange farm, 180 maximum smtp connections into the Orange farm. Setup a round robin DNS A record, say outboundlds.yourdomain.tld pointing to these 6 new IP addresses. At the edge firewall limit the 6 new IPs to outbound TCP 25 traffic only. Configure each of the 6 new Postfix instances as minimally as possible, and to only accept connections from the IP of the primary instance. On the primary instance, you will have a transport_map containing all domains for which you've experienced concurrent delivery problems. orange.fr smtp:outboundlds.yourdomain.tld wanadoo.fr smtp:outboundlds.yourdomain.tld ... smtp:outboundlds.yourdomain.tld ... smtp:outboundlds.yourdomain.tld ... smtp:outboundlds.yourdomain.tld With this setup, you now have essentially an outbound farm of 6 Postfix servers sending from 6 IP addresses to picky receivers. As connection concurrency checking is typically performed by smtpd's on an smtp client IP address basis, and as each of those 10 Orange servers will allow 3 connections per IP address, you will now be able to have 18 concurrent smtp connections to each of the 10 servers in that farm, or 180 total concurrent connections. Currently you can get 30 total connections being they start flogging you. I'd think a peak capacity of 180 concurrent connections should be enough to fix your problem. Hmm, I'll wonder out loud if a check_recipient_mx_access table that returns FILTER slow: would help. It would affect all recipients of a multi-recipient message, but maybe that would be acceptable. I like my idea better. ;) But it is a kludge, and much more complex to implement. I've never done it, so it's possible what I've stated above won't work, and was a big waste of my time combing through postconf5 and Googling. :( It sure would be nice if Postfix would simply round robin outbound smtp connections across all inet_interfaces by default, or have a main.cf option allowing an OP to enable such a thing easily. If such a thing is already in postconf5 I couldn't find it. Using multiple instances to simply accomplish outbound connection balancing across multiple interface IPs is overkill. -- Stan
Re: Re : Re : Re : Re : Re : Re : Re : slow transport, master.cf and maxproc value
On Mon, Mar 07, 2011 at 11:55:34AM -0600, Stan Hoeppner wrote: Unfortunately, the newer postfix rate delay controls are still based on the recipient domain, not the MX destination. Yeah. He can absolutely fix his current problem with it. But if he has to configure huge delays to accomplish it, this obviously creates yet more problems: a backed up queue and mail not delivered in a timely manner. There is a decent chance that once he moves to 2.5 or later, the proposed feedback tuning and larger failure cohort count will solve the issue without rate tuning. The reported issue with the destination is concurrency, not message rate. The OP's mandate to solve the issues with 2.1.5 is rather unfortunate. I'd strongly consider upgrading the legacy systems to 2.7.2 or 2.8.1, after getting a bit of familiarity with the new release on a test machine. -- Viktor.
Re: Postfix 2.8.1 for Solaris
On Mon, Mar 07, 2011 at 06:50:15PM +0100, ??hsan??Do??an wrote: Features: - Native SysV Solaris package - Build with Sun Studio 12.1 - Linked against the native LDAP libraries - No dependencies against 3rd party packages - SMF support - Sparse zone safe - Jumpstart safe The package never updates files in /etc/postfix that are already present. This is only correct for a subset of the files. It would be best to implement the full semantics of postfix-files and only preserve files for which the p flag is set in postfix-files. Other files (makedefs.out, LICENSE, ...) should be updated unconditionally. -- Viktor.
Re: Postfix 2.8.1 for Solaris
Victor Duchovni: On Mon, Mar 07, 2011 at 06:50:15PM +0100, ??hsan??Do??an wrote: Features: - Native SysV Solaris package - Build with Sun Studio 12.1 - Linked against the native LDAP libraries - No dependencies against 3rd party packages - SMF support - Sparse zone safe - Jumpstart safe The package never updates files in /etc/postfix that are already present. This is only correct for a subset of the files. It would be best to implement the full semantics of postfix-files and only preserve files for which the p flag is set in postfix-files. Other files (makedefs.out, LICENSE, ...) should be updated unconditionally. Agreed. You must: - Execute postfix upgrade-configuration after installing postfix. - Respect pathname and mail_owner etc. settings in existing main.cf files. Otherwise, you will produce a broken mail system. - Missing entries will not be added to existing main.cf/master.cf - Files will be installed with the wrong ownership. - Files will be installed in the wrong locations. Wietse
Re: Postfix 2.8.1 for Solaris
On Mon, Mar 07, 2011 at 01:35:13PM -0500, Wietse Venema wrote: You must: - Execute postfix upgrade-configuration after installing postfix. - Respect pathname and mail_owner etc. settings in existing main.cf files. This is done: \$BASEDIR/usr/sbin/chroot \$BASEDIR /usr/sbin/postfix set-permissions upgrade-configuration \ setgid_group=$setgid_group mail_owner=$mail_owner Largely the package looks good, the only nit I found is that all files in /etc/postfix are considered site-specific volatile, which is mostly harmless for now, but the assumption is unwarranted. -- Viktor.
Re: Kernel Oops
Noel Jones put forth on 3/7/2011 9:49 AM:
On 3/7/2011 8:13 AM, Stan Hoeppner wrote:
Noel Jones put forth on 3/7/2011 7:00 AM:
On 3/7/2011 4:47 AM, Stan Hoeppner wrote:
I was taught to always start my expressions with /^ and end them with
$/. Why did Steven teach me to do this if it's not necessary?
That's good advice when you're actually matching something.
Ok, so if I'm doing what I've heard called a fully qualified regular
expression, WRT FQrDNS matching, should I use the anchors or not?
postmap -q says these all work (the actuals with action and text that
is).
/^(\d{1,3}-){3}\d{1,3}\.dynamic\.chello\.sk$/
/^(\d{1,3}\.){4}dsl\.dyn\.forthnet\.gr$/
/^(\d{1,3}-){4}adsl-dyn\.4u\.com\.gh$/
/^[\d\w]{8}\.[\w]{2}-[\d]-[\d\w]{2}\.dynamic\.ziggo\.nl$/
/^(\d{1,3}\.){4}dynamic\.snap\.net\.nz$/
/^pppoe-dyn(-\d{1,3}){4}\.kosnet\.ru$/
In these examples, you're explicitly matching something at the start
and/or end of the string. Using the anchors is correct and recommended.
The special case of .* means, as you know, anything or nothing.
There's never a case where it's necessary to explicitly match a leading
or trailing anything or nothing.
What of the case where you want to match something in the middle of the
input string, with extra junk on both ends?
If you're looking for a string that contains foo anywhere, simply
/foo/
with no anchors.
Consider:
/^.*foo$/
match the string beginning with anything or nothing, ending with foo.
can always be simplified to:
/foo$/
match the string ending with foo.
This works the same without the ending $ anchor (contains foo, rather
than ends with foo), but helps the illustration.
So, in my examples above, given we're matching rDNS patterns, are the
anchors necessary, or helpful? If not using them means contains, then
they should still match. What advantage is there to using the anchors
when matching rDNS patterns? Any?
You use anchors to reduce the chance of a false positive. A side
benefit is improved performance.
Any pattern that matches with the anchors will still match without the
anchors, but may match additional input that you don't intend to match.
In the case of the rDNS patterns, a FP is unlikely (but possible, more
so with the shorter patterns).
In other cases, such as matching a sort bare domain name, a FP may be
very likely without anchors.
best practice is to use the anchors when you can, ie. what you're
matching will always be at the beginning and/or end of the input
string. Never use ^.* or .*$.
Excellent explanations. Thank you Noel.
--
Stan
STARTTLS bug - background story
CERT/CC announces a flaw today in multiple STARTTLS implementations. This problem was silently fixed in Postfix 2.8 and 2.9. Updates for Postfix 2.[4-7] are made available via the usual channels. Wietse Plaintext injection in multiple implementations of STARTTLS === This is a writeup about a flaw that I found recently, and that existed in multiple implementations of SMTP (Simple Mail Transfer Protocol) over TLS (Transport Layer Security) including my Postfix open source mailserver. I give an overview of the problem and its impact, technical background, how to find out if a server is affected, fixes, and draw lessons about where we can expect similar problems now or in the future. A time line is at the end. On-line information is/will be available at: http://www.kb.cert.org/vuls/id/555316 http://www.postfix.org/CVE-2011-0411.html Problem overview and impact === The TLS protocol encrypts communication and protects it against modification by other parties. This protection exists only if a) software is free of flaws, and b) clients verify the server's TLS certificate, so that there can be no man in the middle (servers usually don't verify client certificates). The problem discussed in this writeup is caused by a software flaw. The flaw allows an attacker to inject client commands into an SMTP session during the unprotected plaintext SMTP protocol phase (more on that below), such that the server will execute those commands during the SMTP-over-TLS protocol phase when all communication is supposed to be protected. The injected commands could be used to steal the victim's email or SASL (Simple Authentication and Security Layer) username and password. This is not as big a problem as it may appear to be. The reason is that many SMTP client applications don't verify server TLS certificates. These SMTP clients are always vulnerable to command injection and other attacks. Their TLS sessions are only encrypted but not protected. A similar plaintext injection flaw may exist in the way SMTP clients handle SMTP-over-TLS server responses, but its impact is less interesting than the server-side flaw. SMTP is not the only protocol with a mid-session switch from plaintext to TLS. Other examples are POP3, IMAP, NNTP and FTP. Implementations of these protocols may be affected by the same flaw as discussed here. Technical background: SMTP over TLS === For a precise description of SMTP over TLS, see RFC 3207, on-line at http://www.ietf.org/rfc/rfc3207.txt. SMTP over TLS uses the same TLS protocol that is also used to encrypt traffic between web clients and web servers. But, there is a subtle difference in the way TLS is used, and that makes this flaw possible. SMTP sessions over TLS begin with an SMTP protocol handshake in plaintext. Plaintext means no encryption (thus no privacy), and no protection against modification (no integrity). The plaintext handshake is needed because SMTP has always worked this way. Simply skipping this plaintext phase would seriously break internet email. During the plaintext handshake phase, the SMTP server announces whether it is willing to use TLS. If both SMTP client and server support TLS, the client sends a STARTTLS request to turn on TLS. Once TLS is turned on, all further traffic is encrypted and protected from modification. The client and server repeat the entire SMTP protocol handshake, and the client starts sending mail. Demonstration = The problem is easy to demonstrate with a one-line change to the OpenSSL s_client command source code (I would prefer scripting, but having to install Perl CPAN modules and all their dependencies is more work than downloading a .tar.gz file from openssl.org, adding eight characters to one line, and doing ./config; make). (The OpenSSL s_client command can make a connection to servers that support straight TLS, SMTP over TLS, or a handful other protocols over TLS. The demonstration here focuses on SMTP over TLS only.) The demonstration with SMTP over TLS involves a one-line change in the OpenSSL s_client source code (with OpenSSL 1.0.0, at line 1129 of file apps/s_client.c). Old:BIO_printf(sbio,STARTTLS\r\n); New:BIO_printf(sbio,STARTTLS\r\nRSET\r\n); With this change, the s_client command sends the plaintext STARTTLS command (let's turn on TLS) immediately followed by an RSET command (a relatively harmless protocol reset). Both commands are sent as plaintext in the same TCP/IP packet, and arrive together at the server. The \r\n are the carriage-return and newline characters; these are necessary to terminate an SMTP command. When an SMTP server has the plaintext injection flaw, it reads the STARTTLS command first, switches to SMTP-over-TLS mode, and only then the server reads the RSET command. Note, the RSET command was transmitted during the plaintext SMTP phase when there is
Re : slow transport, master.cf and maxproc value
Hello, I would suggest you grep your mail log for all outbound smtp connections to these 10 IP addresses, and document all the recipient domains. Then add all these domains to your slow transport map. If you still get those 421 errors, completely disable concurrency. If you still get the errors, you probably can't fix the problem until you upgrade to 2.5+ and have access to rate delay controls. Hope this helps. Ok. I'll try to make that. Thank you, -- Myrdhin,
Disable deferred mail sender notification
I've searched but haven't found quite what I'm trying to do. I have found a couple of similar questions here but no answer to my problem. I have a dozen outbound Postfix mail servers (vers. 2.5.5-6.8 2.5.2) processing a relatively large amount of e-mail from a service account, which triggers greylisting from providers such as yahoo.com and txt.att.net. Our inbound server gets overwhelmed processing bounced mail notifications in addition to SPAM and regular inbound e-mail. I was asked to config the outbound mail servers to quit returning deferred message delivery notifications back to our mail server but I'm not really seeing way to do that. These are 4xx messages... usually 421. We still need to process undeliverable (5xx) messages. If I understand correctly, this is handled by the bounce application which is specified in the services section /etc/postfix/master.cf. The line that I believe would need to change is: defer unix - - n - 0 bounce If I understand Postfix, the bounce application will handle both the delivery status notification that I'm trying to suppress and re-queuing the message for later delivery. I don't see any documented options to allow it to re-queue without sending the notification, but I may be reading past it if one is available. I don't see any other mechanism to handle the re-queue without the delivery notification. Can anybody help? I do have and example of the postconf -d if you'd like to see the whole thing, or I could provide specific variables if that would be easier. TIA, Paul
Re: Disable deferred mail sender notification
Paul: I was asked to config the outbound mail servers to quit returning deferred message delivery notifications back to our mail server but I'm not really seeing way to do that. The simplest option is to set delay_warning_time=0. http://www.postfix.org/postconf.5.html#delay_warning_time Changing this takes effect only for new mail. Wietse
Thank you for great software
Hi Wietse I would like to say THANK YOU for postfix because i know developers are hearing this words not often enough as long their baby is running well and nobody takes notice about Especially for the great documentation and crazy online-times on the mailing-list :-) -- Reindl Harald the lounge interactive design GmbH A-1060 Vienna, Hofmühlgasse 17 CTO / software-development / cms-solutions p: +43 (1) 595 3999 33, m: +43 (676) 40 221 40 icq: 154546673, http://www.thelounge.net/ signature.asc Description: OpenPGP digital signature
Re: Pcre header checks
Victor Duchovni wrote: You are lucky this does not work. This is my own private mail server that serves me and my immediate family. If I break stuff everyone on the receiviing end knows who to complain to. Much of the att.net mail infrastructure is operated by Yahoo. Over the last many months, 100% of of the 300+ emails that have a DKIM signaturefrom att.net (yes, even the ones that have a valid DKIM signature and yes, I check it) and came via a yahoo.com mail server have been spam. Given the above data, I think I am justified in using the following pcre rule: /^Received-SPF:.*helo=[a-z0-9.-]+\.mail\\.yahoo\.com; envelope-from=[^@]+@att.net/i REJECT DKIM signatures are also added in messages handled by lists, ... What you are attemtping to do is a bad idea based on a deep misconception of the role of DKIM in email processing. I think I have a fair handle on it. However, my opinion on DKIM is that it is deeply flawed and poorly handled (ie I thing mailing list mangement software should strip DKIM signatures on incoming mail and generate a new DKIM signature on the way out). Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/
Re: Thank you for great software
Reindl Harald: Hi Wietse I would like to say THANK YOU for postfix because i know developers are hearing this words not often enough as long their baby is running well and nobody takes notice about Thanks. I guess this situation is similar to that of a sysadmin when systems are running so well that no-one notices they exist. Especially for the great documentation and crazy online-times on the mailing-list :-) It's the combined on-line presence of many people on this list. My own presence drops a lot in the second and third quarters of the calendar year. Wietse
Re: Pcre header checks
Erik de Castro Lopo wrote: DKIM signatures are also added in messages handled by lists, ... What you are attemtping to do is a bad idea based on a deep misconception of the role of DKIM in email processing. I think I have a fair handle on it. However, my opinion on DKIM is that it is deeply flawed and poorly handled (ie I thing mailing list mangement software should strip DKIM signatures on incoming mail and generate a new DKIM signature on the way out). However, I do admit that my original check (if it had worked) was badly broken. Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/
Re: Kernel Oops
Le 07/03/2011 11:47, Stan Hoeppner a écrit :
mouss put forth on 3/6/2011 7:03 PM:
/^.*foo/
means it starts with something followed by foo. and this is the same
thing as it contains foo, which is represented by
/foo/
I was taught to always start my expressions with /^ and end them with
$/. Why did Steven teach me to do this if it's not necessary? Steven
being the author of the Enemies List: http://enemieslist.com/ which
contains over 65,000 regexes matching FQrDNS patterns.
well, you know I know these:) we all got spam from these...
As with most/all dynamic ranges.
1) first use IP ranges.
2) then domains (hash/cdb)
for example:
.alshamil.net.ae REJECT blah blah
because there is no point to try to match something like
auh-b113917.alshamil.net.ae
3) then use regular expressions, but only when IPs and domains aren't
the way to go.
Well, you know I know these mouss. :)
yes, but we're talking on a public list, so it's good to say it all.
coz' all this stuff is archived and used in way we can't imagine.
Have ever been locked in a
certain train of thought and simply forgot to consider something
related, later putting hand to forehead and saying Duh!. My mindset
was focused on showing how a single PCRE can block the same number of
hosts as using IP addresses in a CIDR or hash table. I just didn't
consider the domain blocking aspect of hash tables at the time. That's
the Duh!. I've been blocking domains with my hash table for something
like 6 years now... I think some folks call this a brain fart. ;)
no. IPs and domains are different things.
cidr is about IPs. hash/cdb/pcre is about names. these are different
things and you know that. use each as appropriate.
Of course. But IPs are valid in a hash table. You can even list them
by the equivalent of a /24, /16, and /8 if you like, simply by omitting
the last 1, 2, or 3 octets of the dotted quad. Just as I brain farted
WRT using domains in a hash table, it appears you have done the same WRT
to using IP addresses in a hash table. :)
not really. I never put IPs in hash tables. more precisely, I never mix
domains and IPs. be it just for the fact that postfix first looks up
domains/hostnames before looking up IPs, which is the opposite of what I
want. the /24, /16, /8 in postfix is a sendmail compat thing.
something I don't need.
I agree it makes more sense to block domains with hash/cdb and IPs with
CIDR. I've been doing exactly that for 5 of the 6 years I've been
running Postfix. The first year (maybe less) I blocked IPs with a hash
table, until I joined this list and learned about CIDR tables. I'm
guessing most other new Postfix OPs go through the same
progression--most beginners docs returned via Google teach the hash
table and nothing else.
if the ISP makes it too much, then you should reduce it:
.embarqhsd.net REJECT blah blah
Yeah, but then you end up potentially blocking large numbers of ham
servers in SOHO land, in this case *.sta.embarqhsd.net. Even in 2011
there are still hundreds of thousands or more SOHO MTAs on static IP
aDSL and cable circuits with generic rDNS. I should know as I'm one of
them. (Please let's not allow this to turn into yet another flame war
WRT generic rDNS, real OPs rent a VPS/colo, yada yada--I'm not directing
this at you mouss but to those predisposed to flog this dead, stripped
to the bone, horse carcass).
believe it or not, I have nothing against dynamic IPs. my approach is
as follows:
- whitelisted IPs get whitelisted. this includes public whitelists and
local whitelists
- I do not include an expression for generic rdns until I get spam
- after N spam, I add an expression. well, I do check if it's ok to add
a blocking rule
- I do not care if it's static, .sta or whatever. as I said above,
it's not about dynamic, it's about accountability. if I get spam from
joe.example, I know I can complain to (abuse|postmaster)@joe.example. if
I get junk from 1.2.3.4.largeisp.example, I know I have no right to
complain, because I'm not part of the money circuit.
a better example would be
/(\W\d+){4}\..*\.embarqhsd\.net$/ REJECT ...
Better in what way?
in the sense that this can't be represented using hash or the like.
Ok. So you're not showing this PCRE above because it better matches the
target rDNS string, or that the engine executes it faster or something,
etc. You're simply saying don't use a PCRE for something you can match
using a simpler table, such as hash/cdb. Correct?
yep. but that said, if you don't have performance problems, using a
single map is probably better than splitting it into a pcre and a
has/cdb map. so what I said doesn't apply to _you_. it was about the
example (showing a better example).
Re: Pcre header checks
On 3/7/2011 4:57 PM, Erik de Castro Lopo wrote: Erik de Castro Lopo wrote: DKIM signatures are also added in messages handled by lists, ... What you are attemtping to do is a bad idea based on a deep misconception of the role of DKIM in email processing. I think I have a fair handle on it. However, my opinion on DKIM is that it is deeply flawed and poorly handled (ie I thing mailing list mangement software should strip DKIM signatures on incoming mail and generate a new DKIM signature on the way out). However, I do admit that my original check (if it had worked) was badly broken. Erik take a look at milter-regexp. -- Noel Jones
regex anchoring (Was: Kernel Oops)
Le 07/03/2011 11:47, Stan Hoeppner a écrit : mouss put forth on 3/6/2011 7:03 PM: /^.*foo/ means it starts with something followed by foo. and this is the same thing as it contains foo, which is represented by /foo/ I was taught to always start my expressions with /^ and end them with $/. Why did Steven teach me to do this if it's not necessary? Steven being the author of the Enemies List: http://enemieslist.com/ which contains over 65,000 regexes matching FQrDNS patterns. You misunderstood what Steven meant. what Stevens meant is to avoid things like /adsl/ REJECT blah so he recommends anchoring expressions, right and left: /^cpe\..*\.joe\.example$/ ... contrast this with /^cpe/ ... and /adsl/ ... which could match a lot of places you wouldn't want to match. /^.*foo/ means: starts with anything followed by foo. this is the same as contains foo, which can be represented by /foo/ and /foo.*$/ means contains foo followed by anything. this is the same as contains foo, which can be represented by /foo/ of course, I appreciate Steven and I agree with what he says here, to some extent (obviously, I'm paid by my employer so it's easy for me to push for freely available stuff). [snip]
Re: Kernel Oops
Le 07/03/2011 15:13, Stan Hoeppner a écrit :
Noel Jones put forth on 3/7/2011 7:00 AM:
On 3/7/2011 4:47 AM, Stan Hoeppner wrote:
I was taught to always start my expressions with /^ and end them with
$/. Why did Steven teach me to do this if it's not necessary?
That's good advice when you're actually matching something.
Ok, so if I'm doing what I've heard called a fully qualified regular
expression, WRT FQrDNS matching, should I use the anchors or not?
postmap -q says these all work (the actuals with action and text that is).
/^(\d{1,3}-){3}\d{1,3}\.dynamic\.chello\.sk$/
.dynamic.chello.sk REJECT blah blah
/^(\d{1,3}\.){4}dsl\.dyn\.forthnet\.gr$/
.dyn.forthnet.grREJECT blah blah
/^(\d{1,3}-){4}adsl-dyn\.4u\.com\.gh$/
/dyn\.4u.com\.gh$/ REJECT blah
assuming you get real mail from there. otherwise
.4u.com.gh REJECT blah
/^[\d\w]{8}\.[\w]{2}-[\d]-[\d\w]{2}\.dynamic\.ziggo\.nl$/
ahem? I fail to see what yoy're trying to match here. \d is a \w, so
[\d\w] is the same as \w. do you mean \W (capital letter)? anyway:
.dynamic.ziggo.nlREJECT blah blah
/^(\d{1,3}\.){4}dynamic\.snap\.net\.nz$/
.dynamic.snap.net.nzREJECT blah
/^pppoe-dyn(-\d{1,3}){4}\.kosnet\.ru$/
/\Wdyn\W.*\.kosnet\.ru$/REJECT blah
The special case of .* means, as you know, anything or nothing.
There's never a case where it's necessary to explicitly match a leading
or trailing anything or nothing.
What of the case where you want to match something in the middle of the
input string, with extra junk on both ends?
well, that's what regular expressions are about by default:
/foo/ means contains foo
/^foo/ means starts with foo
/foo$/ means ends with foo
so
/^bart.*homer.*marge$/ means: starts with bart, ends with marge and
somewhere between these contains homer.
Consider:
/^.*foo$/
match the string beginning with anything or nothing, ending with foo.
can always be simplified to:
/foo$/
match the string ending with foo.
This works the same without the ending $ anchor (contains foo, rather
than ends with foo), but helps the illustration.
So, in my examples above, given we're matching rDNS patterns, are the
anchors necessary, or helpful? If not using them means contains, then
they should still match. What advantage is there to using the anchors
when matching rDNS patterns? Any?
(In the other special case where you're using $1, $2, etc. substitution
in the result, you might need some form of /^(.*foo)$/ to fill the
substitution buffer, but that's about substitution, not about matching.)
Thank you for the continuing PCRE education Noel, and Ansgar. :)
Re: Kernel Oops
it is necessary to consider the option
parent_domain_matches_subdomains =
Le mardi 08 mars 2011 à 00:45 +0100, mouss a écrit :
Le 07/03/2011 15:13, Stan Hoeppner a écrit :
Noel Jones put forth on 3/7/2011 7:00 AM:
On 3/7/2011 4:47 AM, Stan Hoeppner wrote:
I was taught to always start my expressions with /^ and end them with
$/. Why did Steven teach me to do this if it's not necessary?
That's good advice when you're actually matching something.
Ok, so if I'm doing what I've heard called a fully qualified regular
expression, WRT FQrDNS matching, should I use the anchors or not?
postmap -q says these all work (the actuals with action and text that is).
/^(\d{1,3}-){3}\d{1,3}\.dynamic\.chello\.sk$/
.dynamic.chello.skREJECT blah blah
/^(\d{1,3}\.){4}dsl\.dyn\.forthnet\.gr$/
.dyn.forthnet.gr REJECT blah blah
/^(\d{1,3}-){4}adsl-dyn\.4u\.com\.gh$/
/dyn\.4u.com\.gh$/REJECT blah
assuming you get real mail from there. otherwise
.4u.com.ghREJECT blah
/^[\d\w]{8}\.[\w]{2}-[\d]-[\d\w]{2}\.dynamic\.ziggo\.nl$/
ahem? I fail to see what yoy're trying to match here. \d is a \w, so
[\d\w] is the same as \w. do you mean \W (capital letter)? anyway:
.dynamic.ziggo.nl REJECT blah blah
/^(\d{1,3}\.){4}dynamic\.snap\.net\.nz$/
.dynamic.snap.net.nz REJECT blah
/^pppoe-dyn(-\d{1,3}){4}\.kosnet\.ru$/
/\Wdyn\W.*\.kosnet\.ru$/ REJECT blah
The special case of .* means, as you know, anything or nothing.
There's never a case where it's necessary to explicitly match a leading
or trailing anything or nothing.
What of the case where you want to match something in the middle of the
input string, with extra junk on both ends?
well, that's what regular expressions are about by default:
/foo/ means contains foo
/^foo/ means starts with foo
/foo$/ means ends with foo
so
/^bart.*homer.*marge$/ means: starts with bart, ends with marge and
somewhere between these contains homer.
Consider:
/^.*foo$/
match the string beginning with anything or nothing, ending with foo.
can always be simplified to:
/foo$/
match the string ending with foo.
This works the same without the ending $ anchor (contains foo, rather
than ends with foo), but helps the illustration.
So, in my examples above, given we're matching rDNS patterns, are the
anchors necessary, or helpful? If not using them means contains, then
they should still match. What advantage is there to using the anchors
when matching rDNS patterns? Any?
(In the other special case where you're using $1, $2, etc. substitution
in the result, you might need some form of /^(.*foo)$/ to fill the
substitution buffer, but that's about substitution, not about matching.)
Thank you for the continuing PCRE education Noel, and Ansgar. :)
--
gpg --keyserver pgp.mit.edu --recv-key 092164A7
http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x092164A7
signature.asc
Description: Ceci est une partie de message numériquement signée
Re: STARTTLS bug - background story
On Tue, 8 Mar 2011 07:08:09 am Wietse Venema wrote: This is a writeup about a flaw that I found recently, and that existed in multiple implementations of SMTP (Simple Mail Transfer Protocol) over TLS (Transport Layer Security) including my Postfix open source mailserver. I give an overview of the problem and its impact, technical background, how to find out if a server is affected, fixes, and draw lessons about where we can expect similar problems now or in the future. A time line is at the end. Thanks for the write-up. Brad
Re: Pcre header checks
Noel Jones wrote: take a look at milter-regexp. Thanks, I'll check it out. Cheers, Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/
Re: STARTTLS bug - background story
On Tue, Mar 08, 2011 at 12:59:15PM +1100, Brad Hards wrote: On Tue, 8 Mar 2011 07:08:09 am Wietse Venema wrote: This is a writeup about a flaw that I found recently, and that existed in multiple implementations of SMTP (Simple Mail Transfer Protocol) over TLS (Transport Layer Security) including my Postfix open source mailserver. I give an overview of the problem and its impact, technical background, how to find out if a server is affected, fixes, and draw lessons about where we can expect similar problems now or in the future. A time line is at the end. Thanks for the write-up. It is a bit disappointing that very few of the potentially impacted vendors, and some definitely impacted vendors are yet to respond to the vulnerability: http://www.kb.cert.org/vuls/id/555316 Some email appliance vendors are not on the list. Apart from Postfix, Qmail, and some large mailbox hosting providers, which are already fixed, the issue will likely linger in less visible products for some time... -- Viktor.
Re: Disable deferred mail sender notification
Wietse, thanks for the quick response, and a solution that looks like it might work. That wouldn't also suppress 5xx bounce notifications, would it? In order to be good e-mail citizens we have to capture those undeliverable addresses and remove them from our active mail database. I'm assuming that the line time after which the sender receives the message headers of mail that is still queued implies that hard bounces would be removed from the queue and we would still get the bounce notification. Thanks again for your help. Paul Paul: I was asked to config the outbound mail servers to quit returning deferred message delivery notifications back to our mail server but I'm not really seeing way to do that. The simplest option is to set delay_warning_time=0. http://www.postfix.org/postconf.5.html#delay_warning_time Changing this takes effect only for new mail. Wietse
Re: Thank you for great software
On Mon, 07 Mar 2011 23:47:30 +0100 Reindl Harald h.rei...@thelounge.net wrote: Especially for the great documentation Not to mention this list and the vast knowledge all the subscribers have to offer. It has been invaluable.
Re: smtpd_sasl_path tcp-socket?
Hello, smtpd_sasl_path = inet:localhost:1434 seems to work but it is not documented. As documented: smtpd_sasl_path (default: smtpd) Implementation-specific information that the Postfix SMTP server passes through to the SASL plug-in implementation that is selected with smtpd_sasl_type. So, you are asking your question in the wrong place. hmm, Timo Sirainen did sent me back. ;) http://dovecot.org/list/dovecot/2011-March/057773.html src/xsasl/xsasl_dovecot_server.c was originally written by him but by now heavily modified. when SASL type is dovecot, SASL path inet:localhost:1434 should be a socket pathname this warning is no longer required. may be an advice in readme that this auth-traffic is unsecured... Thanks, Hajo
