Aggregating/rate-limiting emails
Are there any extensions to Postfix that can aggregate multiple outgoing emails into a single email within some time window? We're developing an application that runs on multiple hosts and emails notifications to us (the developers @gmail.com) whenever something goes wrong, via a postfix server. However, we've run into issues where the application spews hundreds of such errors in rapid succession, leading Gmail to bounce our messages. This is why we're interested first and foremost throttling messages, but ideally also aggregating messages together into a periodic digest that is emitted at most once per minute. Any other (low-effort) solution ideas would be appreciated as well. -- Yang Zhang http://yz.mit.edu/
Re: Aggregating/rate-limiting emails
* Yang Zhang yanghates...@gmail.com: Are there any extensions to Postfix that can aggregate multiple outgoing emails into a single email within some time window? Not that I'm aware of. You're thinking of something like a mailing-list digest? We're developing an application that runs on multiple hosts and emails notifications to us (the developers @gmail.com) whenever something goes wrong, via a postfix server. However, we've run into issues where the application spews hundreds of such errors in rapid succession, leading Gmail to bounce our messages. This is why we're interested first and foremost throttling messages, That's easy: either via a policy server OR you use something like smtp_destination_rate_delay = 10s (one mail very 10s) but ideally also aggregating messages together into a periodic digest that is emitted at most once per minute. Any other (low-effort) solution ideas would be appreciated as well. -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Aggregating/rate-limiting emails
On 09/08/2010 12:33 PM, Yang Zhang wrote: Are there any extensions to Postfix that can aggregate multiple outgoing emails into a single email within some time window? We're developing an application that runs on multiple hosts and emails notifications to us (the developers @gmail.com) whenever something goes wrong, via a postfix server. However, we've run into issues where the application spews hundreds of such errors in rapid succession, leading Gmail to bounce our messages. This is why we're interested first and foremost throttling messages, but ideally also aggregating messages together into a periodic digest that is emitted at most once per minute. Any other (low-effort) solution ideas would be appreciated as well. -- Yang Zhang http://yz.mit.edu/ how about writing the errors to a log and email the log via a cron job ?
Re: timeout trouble with postfix and amavisd in BQCF
On 3 sept. 2010, at 19:49, Mark Martinec wrote: Machine does not look busy at all during those problems. Load is under 0.5 and CPU is 90% idle. Even small emails are affected. If the host is not busy, again, my primary suspect is a berkeley db. These multiples of 20..25 second delays, some at inexplicable sections, seem to coincide with updating a child process status in the nanny database. Try disabling it altogether: $enable_db=0; If that helps, consider upgrading libdb to a more recent version (along with the BerkeleyDB perl module). enable_db=0 does the trick, for now. I've found this about amavisd and BDB performance problem on FreeBSD: http://www.mail-archive.com/amavis-u...@lists.sourceforge.net/msg15381.html I'll test ASAP. regards, Patrick PRONIEWSKI -- Administrateur Système - SENTIER - Université Lumière Lyon 2 smime.p7s Description: S/MIME cryptographic signature
Re: Aggregating/rate-limiting emails
On Wed, Sep 8, 2010 at 12:16 AM, Mihira Fernando mihirathe...@gmail.com wrote: On 09/08/2010 12:33 PM, Yang Zhang wrote: Are there any extensions to Postfix that can aggregate multiple outgoing emails into a single email within some time window? We're developing an application that runs on multiple hosts and emails notifications to us (the developers @gmail.com) whenever something goes wrong, via a postfix server. However, we've run into issues where the application spews hundreds of such errors in rapid succession, leading Gmail to bounce our messages. This is why we're interested first and foremost throttling messages, but ideally also aggregating messages together into a periodic digest that is emitted at most once per minute. Any other (low-effort) solution ideas would be appreciated as well. -- Yang Zhang http://yz.mit.edu/ how about writing the errors to a log and email the log via a cron job ? We were actually considering just tailing our logs for errors, but it's a bit more work because a bunch of the error reporting logic is actually not necessarily logged, and the errors are caught/emails are generated by libraries that we didn't develop. Failing to find other solutions we might opt for this. -- Yang Zhang http://yz.mit.edu/
Re: Aggregating/rate-limiting emails
On Wed, Sep 8, 2010 at 12:15 AM, Ralf Hildebrandt ralf.hildebra...@charite.de wrote: * Yang Zhang yanghates...@gmail.com: This is why we're interested first and foremost throttling messages, That's easy: either via a policy server OR you use something like smtp_destination_rate_delay = 10s (one mail very 10s) I failed to mention that our mail server *does* actually need to send other emails normally - only emails from err...@ourdomain.com should be rate-limited. -- Yang Zhang http://yz.mit.edu/
Re: reject_unknown_client_hostname light?
On 2010-09-08 06:02, pf at alt-ctrl-del.org wrote: Am I missing something obvious? With many ISPs providing generic PTR, reject_unknown_reverse_client_hostname is too gentle. I'd really like to implement reject_unknown_client_hostname, but I've seen too many cases where address-name mapping = exists, the name-address mapping = exists, BUT the name-address mapping is in the same /24 - but off by a couple of IPs. Is there a test that I'm missing out on that is simply address-name mapping = exists, and name-any address mapping = exists? Or a chain of tests that can accomplish the same thing? You can use a policy server for implementing more complicated restrictions than Postfix supports internally. Postfwd http://www.postfwd.org has a very flexible configuration syntax and can probably do what you want. You could also write your own custom policy server. See also http://www.postfix.org/SMTPD_POLICY_README.html
Re: set envelope sender = sasl authenticated user ?
On Tue, Sep 07, 2010 at 06:38:15PM -0500, Noel Jones wrote: If you have customers sending large amounts of abusive mail, seems as if there would be better ways to deal with that eg. sender quotas, monitoring of undeliverable mail, inbound spam/virus scanning, etc. But I'm not an ISP; I can fire anyone who abuses the mail system. We do sender quotas, some monitoring of undeliverable mail, in and outbound spam/virus scanning and more, but this doesn't catch all. Users that are clearly abusing the system (read: has malware installed) gets blocked automatically or manually, but there's a time window where they will be able to send out junk, and when you have enough customers -- someone will always have the latest and greatest malware installed and we woun't catch it immediately. and I still fail to understand how controlling your customers envelope sender will help with backscatterer.org. It will make sure that when viruses/malware on the customers computer is sending out spam from fake addresses, the bounces goes back to the customer with the infected computer -- instead of to whomever the malware was pretending to send from. -jf
Re: Aggregating/rate-limiting emails
Le Wed, 8 Sep 2010 00:29:03 -0700, Yang Zhang yanghates...@gmail.com a écrit : On Wed, Sep 8, 2010 at 12:16 AM, Mihira Fernando mihirathe...@gmail.com wrote: On 09/08/2010 12:33 PM, Yang Zhang wrote: Are there any extensions to Postfix that can aggregate multiple outgoing emails into a single email within some time window? We're developing an application that runs on multiple hosts and emails notifications to us (the developers @gmail.com) whenever something goes wrong, via a postfix server. However, we've run into issues where the application spews hundreds of such errors in rapid succession, leading Gmail to bounce our messages. This is why we're interested first and foremost throttling messages, but ideally also aggregating messages together into a periodic digest that is emitted at most once per minute. Any other (low-effort) solution ideas would be appreciated as well. -- Yang Zhang http://yz.mit.edu/ how about writing the errors to a log and email the log via a cron job ? We were actually considering just tailing our logs for errors, but it's a bit more work because a bunch of the error reporting logic is actually not necessarily logged, and the errors are caught/emails are generated by libraries that we didn't develop. Failing to find other solutions we might opt for this. -- Yang Zhang http://yz.mit.edu/ A bit twisted: you could send to a local mailbox on the postfix server, then use procmail or similar via cron to send digests. Seems to me that procmail can do this (multiple bodies feed a new mail). You may also set up a transport for these mails, and a particular transport_destination_rate_delay Xavier Gillard
sender and recipient dependend routing in a single postfix engine
Hello @all I wonder if there is a possibility to tell postfix how to route e-mails sender and recipient dependent. Our scenario is as follows. Postfix is anti spam and anti virus gateway for a couple of domains. Incoming mail for all these domains is forwarded by the Provider to this gateway and distributed to several internal mail servers (exchange, domino) in respect to the transport table. For sending e-mail all internal mail servers use the gateway as a smart host. Sending mail to each other works as well as sending mail to external recipients. Additionally there is a special requirement to send e-mail for a given list of domains through another gateway and not the provider's. This is done by using a second transport table. Now the question: Is it possible to have an additional mechanism to alter the second routing decision in respect to the senders address? kind regards Ilja
Re: sender and recipient dependend routing in a single postfix engine
Am 08.09.2010 11:36, schrieb Ilja Beeskow: Hello @all I wonder if there is a possibility to tell postfix how to route e-mails sender and recipient dependent. Our scenario is as follows. Postfix is anti spam and anti virus gateway for a couple of domains. Incoming mail for all these domains is forwarded by the Provider to this gateway and distributed to several internal mail servers (exchange, domino) in respect to the transport table. For sending e-mail all internal mail servers use the gateway as a smart host. Sending mail to each other works as well as sending mail to external recipients. Additionally there is a special requirement to send e-mail for a given list of domains through another gateway and not the provider's. This is done by using a second transport table. Now the question: Is it possible to have an additional mechanism to alter the second routing decision in respect to the senders address? kind regards Ilja http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps should help -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: sender and recipient dependend routing in a single postfix engine
Ilja Beeskow: Hello @all I wonder if there is a possibility to tell postfix how to route e-mails sender and recipient dependent. Our scenario is as follows. You may have to use an SMTPD policy daemon that replies with FILTER transport:nexthop. Not every mail routing problem can be solved with Postfix built-ins. Wietse
Re: Aggregating/rate-limiting emails
Yang Zhang: Are there any extensions to Postfix that can aggregate multiple outgoing emails into a single email within some time window? We're developing an application that runs on multiple hosts and emails notifications to us (the developers @gmail.com) whenever something goes wrong, via a postfix server. However, we've run into issues where the application spews hundreds of such errors in rapid succession, leading Gmail to bounce our messages. This is why we're interested first and foremost throttling messages, but ideally also aggregating messages together into a periodic digest that is emitted at most once per minute. Any other (low-effort) solution ideas would be appreciated as well. Aggregate at the SOURCE: append all alerts to a file. Use a once-per-minute cron job to rename the file and send out the alerts. Wietse
Re: Aggregating/rate-limiting emails
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Wietse Venema wrote: | Yang Zhang: | Are there any extensions to Postfix that can aggregate multiple | outgoing emails into a single email within some time window? 8-8 | aggregating messages together into a periodic digest that is emitted | at most once per minute. Any other (low-effort) solution ideas would | be appreciated as well. | | Aggregate at the SOURCE: append all alerts to a file. Use a | once-per-minute cron job to rename the file and send out the alerts. | | Wietse Another idea: use a mailing list daemon with the ability to send digest messages based on volume (i.e.: send the digest message when there have been N messages received by the list) like Mailman or Sympa. - -- Victoriano Giralt Systems Manager Central ICT Services University of Malaga SPAIN - - A: Yes. | Q: Are you sure ? | A: Because it reverses the logical flow of conversation. | Q: Why is top posting annoying in email ? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with CentOS - http://enigmail.mozdev.org iD8DBQFMh3KlV6+mDjj1PTgRAjakAKDJh4A90QQhVJGbzT61bO0nJtzfCgCfcfhv pKmXUCnX4HUk3SvyAV1j64I= =uz1K -END PGP SIGNATURE-
Re: Aggregating/rate-limiting emails
Yang Zhang: Are there any extensions to Postfix that can aggregate multiple outgoing emails into a single email within some time window? ... aggregating messages together into a periodic digest that is emitted at most once per minute. Any other (low-effort) solution ideas would be appreciated as well. Wietse: Aggregate at the SOURCE: append all alerts to a file. Use a once-per-minute cron job to rename the file and send out the alerts. This will send mail once per minute as long as there was an alert. Victoriano Giralt: Another idea: use a mailing list daemon with the ability to send digest messages based on volume (i.e.: send the digest message when there have been N messages received by the list) like Mailman or Sympa. This will not send any mail at all while there are less than N alerts. Wietse
Re: reject_unknown_client_hostname light?
pf at alt-ctrl-del.org: Am I missing something obvious? Yes. http://www.postfix.org/postconf.5.html#reject_unknown_reverse_client_hostname Wietse
Re: sender and recipient dependend routing in a single postfix engine
Am 08.09.2010 12:40, schrieb Robert Schetterer: Am 08.09.2010 11:36, schrieb Ilja Beeskow: Hello @all I wonder if there is a possibility to tell postfix how to route e-mails sender and recipient dependent. Our scenario is as follows. [...] Ilja http://www.postfix.org/postconf.5.html#sender_dependent_default_transport_maps should help Thank you Robert I found that too but it seemed to help only at first glance: From postconf.5.html: 'Note: this overrides default_transport, not transport_maps' Greetings Ilja
Re: sender and recipient dependend routing in a single postfix engine
Am 08.09.2010 13:13, schrieb Wietse Venema: Ilja Beeskow: Hello @all I wonder if there is a possibility to tell postfix how to route e-mails sender and recipient dependent. Our scenario is as follows. You may have to use an SMTPD policy daemon that replies with FILTER transport:nexthop. Not every mail routing problem can be solved with Postfix built-ins. Wietse Hello Wietse I suspected this! Thank you for your tip - perhaps we try it later. For now the feature is 'not available'! ;-) Greetings Ilja
Re: Can postfix work with a TLS, authenticated smtp relay server?
Richard Chapman wrote: Perhaps you are describing an alternative method for google apps smtp which I am unaware of. If so - can you point me to a description of this alternative option? I fail to see why you need postfix if your domain is hosted on Google Apps. Google Apps provide you with Webmail, IMAP4, and SMTP (submission). https://www.google.com/a/ http://mail.google.com/support/a/google.com/bin/answer.py?answer=33384 HTH, Mikael
Re: set envelope sender = sasl authenticated user ?
Jan-Frode Myklebust wrote: and I still fail to understand how controlling your customers envelope sender will help with backscatterer.org. It will make sure that when viruses/malware on the customers computer is sending out spam from fake addresses, the bounces goes back to the customer with the infected computer -- instead of to whomever the malware was pretending to send from. I have never seen malware use SMTP AUTH via the smarthost SMTP. Most malware shoot directly on 25/tcp. Maybe you'll be better off blocking 25/tcp and force users to use the submission port (587/tcp) with SMTP AUTH and possibly STARTTLS. I think your spam problems will go away if you do that. HTH, Mikael
Re: integrate dspam into postfix
Is there a way to use virtual_tranport with virtual_alias for this case?
On Sep 3, 2010, at 1:35 AM, Martijn de Munnik wrote:
Hi list,
I'm trying to integrate dspam filtering into my postfix system. The
way I have it now works for local users but when a user has an alias
to an external domain the mail bounces. This server is for receiving
mail only, so no submission is needed.
---
This is the mail system at host chuck.redknot.nl.
I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can
delete your own text from the attached returned message.
The mail system
martijndemun...@chuck.redknot.nl (expanded from
martijndemunnik): unknown
user: martijndemunnik
Reporting-MTA: dns; chuck.redknot.nl
X-Postfix-Queue-ID: 1C455D2A2
X-Postfix-Sender: rfc822; mart...@youngguns.nl
Arrival-Date: Thu, 2 Sep 2010 21:58:12 +0200 (CEST)
Final-Recipient: rfc822; martijndemun...@chuck.redknot.nl
Original-Recipient: rfc822;martijndemunnik
Action: failed
Status: 5.1.1
Diagnostic-Code: X-Postfix; unknown user: martijndemunnik
---
The virtual file contains:
just...@suezkade.nl martijndemun...@gmail.com
I'm sure this is because I pass --user ${mailbox} to the dspam
command, but I'm not sure how to solve this. I want dspam to learn
what is spam for my local user, so I guess dspam should be as close
as possible to final delivery. There is also another problem. Dspam
calls clamav to scan the message for viruses. When A virus is found
dspam dies because the shell can't handle a negative error return
code. The dspam list told me to use the server part of dspam to fix
this and let postfix talk to dspam with lmtp. I'm not sure how to do
this. virtual_transport doesn't seem to have any effect because I'm
not using virtual_domains?
Any ideas?
Thanks,
Martijn
Output of postconf -n:
address_verify_map = btree:${data_directory}/verify
alias_maps = dbm:/etc/opt/redknot/postfix/aliases
config_directory = /etc/opt/redknot/postfix
content_filter = dspam:dpsam
disable_vrfy_command = yes
home_mailbox = Maildir/
mailbox_command = /opt/redknot/libexec/dovecot/deliver -a
$RECIPIENT -m $EXTENSION -s
mydestination = $myhostname, localhost.$mydomain, localhost
mydomain = chuck.redknot.nl
myhostname = chuck.redknot.nl
recipient_delimiter = +
relay_domains = $mydestination, atdstramproy.nl
smtpd_banner = $myhostname ESMTP
smtpd_data_restrictions = reject_unauth_pipelining, permit
smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_proxy_filter = 127.0.0.1:10027
smtpd_proxy_options = speed_adjust
smtpd_recipient_restrictions = reject_non_fqdn_helo_hostname,
reject_non_fqdn_sender, reject_non_fqdn_recipient,
reject_unknown_sender_domain, reject_unknown_recipient_domain,
reject_unauth_destination, reject_rbl_client virbl.dnsbl.bit.nl,
check_policy_service inet:localhost:10031, check_policy_service
inet:localhost:10030, check_policy_service inet:localhost:10029,
permit
smtpd_tls_cert_file = /etc/opt/redknot/ssl/chuck.redknot.nl.cer
smtpd_tls_key_file = /etc/opt/redknot/ssl/chuck.redknot.nl.key
smtpd_use_tls = yes
soft_bounce = yes
strict_rfc821_envelopes = yes
transport_maps = dbm:/etc/opt/redknot/postfix/transport
unknown_address_reject_code = 550
unknown_hostname_reject_code = 550
unknown_local_recipient_reject_code = 550
unverified_recipient_reject_code = 550
virtual_alias_maps = dbm:/etc/opt/redknot/postfix/virtual
and my master.cf
#
# Postfix master process configuration file. For details on the
format
# of the file, see the master(5) manual page (command: man 5
master).
#
# Do not forget to execute postfix reload after editing this file.
#
#
=
=
=
=
==
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
#
=
=
=
=
==
smtp inet n - n - - smtpd
#submission inet n - n - - smtpd
# -o smtpd_tls_security_level=encrypt
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
#smtps inet n - n - - smtpd
# -o smtpd_tls_wrappermode=yes
# -o smtpd_sasl_auth_enable=yes
# -o smtpd_client_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
pickupfifo n - n 60 1 pickup
cleanup unix n - n - 0 cleanup
qmgr fifo n - n 300 1 qmgr
tlsmgrunix - - n 1000? 1 tlsmgr
rewrite unix - - n - - trivial-
Re: set envelope sender = sasl authenticated user ?
On 09/08/2010 12:36 AM, Jan-Frode Myklebust wrote: On Tue, Sep 07, 2010 at 08:20:36PM +0200, Jeroen Geilman wrote: On 09/07/2010 06:57 PM, mouss wrote: OP is an ISP providing outbound relay to residential users. his problem is not easy to solve. Thanks for understanding. I´ve gotten information off list that gmail are setting the sender to the gmail authenticated user when sending from non-gmail addresses. So I´m at least not alone in thinking this should be a good solution. Also I believe you have to pre-register any address you want to send from trough gmail, which sounds like we could use postfix´ smtpd_sender_login_maps to implement a similar solution. Residential users don't often have their own mail servers/domains. Limiting these to sending with their true ISP address is fairly common. (And just about the only sane way to implement this kind of relay) In Norway all/most ISPs are forcing their residential users trough the ISP`s smarthosts. Outgoing port 25/tcp is blocked. So users are not allowed to run their own mailservers. I can´t justify also requiring all users to use the ISP´s mail addresses. -jf Hi, iam running a small ISP here in egypt and we are using smtpd_sender_login maps to prevent anyone unauthenticated to send from email addresses they don't own, it works very well and postifix responds with an error address not owned by user. one challenge we faced was that the database query has to include all aliases which are registered for the authenticated username (in our case the email) but this is no problem with some smart views on your database table. so set smtpd_sender_login_maps = proxy:pgsql:/path.cf and then in smtpd_sender_restrictions = reject_sender_login_mismatch and you are done. PS: you should of course separate the incoming client mail and the domain domain mail transfers, so for example mailserver to mailserver uses port 25 where this policy is not applied and clients use port 587 submission where this policy is applied. Frank
Re: integrate dspam into postfix
Hi List,
I'm still struggling with dspam integration with postfix.
Now I have:
--
address_verify_map = btree:${data_directory}/verify
alias_maps = dbm:/etc/opt/redknot/postfix/aliases
config_directory = /etc/opt/redknot/postfix
disable_vrfy_command = yes
home_mailbox = Maildir/
mailbox_command = /opt/redknot/libexec/dovecot/deliver -a $RECIPIENT
-m $EXTENSION -s
mailbox_transport = dspam-lmtp:[127.0.0.1]:10025
mydestination = $myhostname, localhost.$mydomain, localhost
mydomain = chuck.redknot.nl
myhostname = chuck.redknot.nl
recipient_delimiter = +
relay_domains = $mydestination, atdstramproy.nl
smtpd_banner = $myhostname ESMTP
smtpd_data_restrictions = reject_unauth_pipelining, permit
smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_proxy_filter = 127.0.0.1:10027
smtpd_proxy_options = speed_adjust
smtpd_recipient_restrictions = reject_non_fqdn_helo_hostname,
reject_non_fqdn_sender, reject_non_fqdn_recipient,
reject_unknown_sender_domain, reject_unknown_recipient_domain,
reject_unauth_destination, reject_rbl_client virbl.dnsbl.bit.nl,
check_policy_service inet:localhost:10031, check_policy_service
inet:localhost:10030, check_policy_service inet:localhost:10029,
permit
smtpd_tls_cert_file = /etc/opt/redknot/ssl/chuck.redknot.nl.cer
smtpd_tls_key_file = /etc/opt/redknot/ssl/chuck.redknot.nl.key
smtpd_use_tls = yes
soft_bounce = yes
strict_rfc821_envelopes = yes
transport_maps = dbm:/etc/opt/redknot/postfix/transport
unknown_address_reject_code = 550
unknown_hostname_reject_code = 550
unknown_local_recipient_reject_code = 550
unverified_recipient_reject_code = 550
virtual_alias_maps = dbm:/etc/opt/redknot/postfix/virtual
--
So I'm using a mailbox_transport to call dspam. Unfortantly the mail
doesn't show up in the logs after the lmtp part (I have lmtp -v in
master.cf) and the mail isn't delivered. When I remove the
mailbox_transport and restart postfix the mails get delivered. Why
does the LMTP conversation stop in the last three lines, I expect to
see the actual message?
Sep 8 16:38:32 chuck postfix/smtpd[24691]: [ID 197553 mail.info]
connect from mail-yw0-f44.google.com[209.85.213.44]
Sep 8 16:38:38 chuck policyd-spf[24698]: [ID 702911 mail.info] None;
identity=helo; client-ip=209.85.213.44; helo=mail-yw0-f44.google.com; envelope-from=martijndemun...@gmail.com
; receiver=mart...@redknot.nl
Sep 8 16:38:38 chuck policyd-spf[24698]: [ID 702911 mail.info] Pass;
identity=mailfrom; client-ip=209.85.213.44; helo=mail-yw0-
f44.google.com; envelope-from=martijndemun...@gmail.com; receiver=mart...@redknot.nl
Sep 8 16:38:38 chuck postfix/smtpd[24691]: [ID 197553 mail.info]
NOQUEUE: client=mail-yw0-f44.google.com[209.85.213.44]
Sep 8 16:38:39 chuck postfix/smtpd[24686]: [ID 197553 mail.info]
connect from localhost[127.0.0.1]
Sep 8 16:38:39 chuck postfix/smtpd[24686]: [ID 197553 mail.info]
167E810897: client=mail-yw0-f44.google.com[209.85.213.44]
Sep 8 16:38:39 chuck postfix/cleanup[24687]: [ID 197553 mail.info]
167E810897: message-id=aanlkti=5jghf56pzvnfr0qqhvxqwk_zvxndx18eox...@mail.gmail.com
Sep 8 16:38:39 chuck postfix/qmgr[24585]: [ID 197553 mail.info]
167E810897: from=martijndemun...@gmail.com, size=2401, nrcpt=1
(queue active)
Sep 8 16:38:39 chuck postfix/smtpd[24691]: [ID 197553 mail.info]
proxy-accept: END-OF-MESSAGE: 250 2.0.0 Ok: queued as 167E810897;
from=martijndemun...@gmail.com to=mart...@redknot.nl proto=ESMTP
helo=mail-yw0-f44.google.com
Sep 8 16:38:39 chuck postfix/smtpd[24686]: [ID 197553 mail.info]
disconnect from localhost[127.0.0.1]
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: const mail
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: const ipv4
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: const
Sep 8 16:38:39 chuck last message repeated 2 times
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
name_mask: ipv4
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: const chuck.redknot.nl
Sep 8 16:38:39 chuck last message repeated 1 time
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: const Postfix
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: expand ${multi_instance_name:postfix}${multi_instance_name?
$multi_instance_name} - postfix
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: const postfix
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: const postdrop
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: expand $myhostname, localhost.$mydomain, localhost -
chuck.redknot.nl, localhost.chuck.redknot.nl, localhost
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: expand $myhostname - chuck.redknot.nl
Sep 8 16:38:39 chuck postfix/lmtp[24700]: [ID 197553 mail.info]
dict_eval: const
Sep 8
Re: integrate dspam into postfix
Martijn de Munnik: So I'm using a mailbox_transport to call dspam. Unfortantly the mail doesn't show up in the logs after the lmtp part (I have lmtp -v in master.cf) and the mail isn't delivered. When I remove the The mailbox_transport delivers the mail to dspam, therefore the mail no longer exists in the Postfix mail queue. I suggest that you have a look at the Postfix FILTER_README documentation. Wietse
Re: integrate dspam into postfix
On Sep 8, 2010, at 5:34 PM, Wietse Venema wrote: Martijn de Munnik: So I'm using a mailbox_transport to call dspam. Unfortantly the mail doesn't show up in the logs after the lmtp part (I have lmtp -v in master.cf) and the mail isn't delivered. When I remove the The mailbox_transport delivers the mail to dspam, therefore the mail no longer exists in the Postfix mail queue. But the mails are still listed when I issue a mailq and when I remove the mailbox_transport line from main.cf and restart postfix the mails are delivered immediately. DSPAM should reinsert the message into the postfix queue after processing. This works when I use a content_filter and call dspam using a pipe. I suggest that you have a look at the Postfix FILTER_README documentation. Will do. Wietse
Re: Can postfix work with a TLS, authenticated smtp relay server?
On Wed, Sep 08, 2010 at 11:12:45AM +0800, Richard Chapman wrote: AFAIK smtp.google.com requires an authenticated TLS connection. If you have a Google Apps hosted domain, you use fixed MTA credentials, (possibly just an IP whitelist) negotiated with Google and send to alternate servers (not smtp.gmail.com 587). No per-user credentials required. Not sure I fully understand this... I AM using a google apps hosted domain - and I think I need to use per user credentials because that is what would happen if the users connected direct to smtp.google.com as described in the google apps setup instructions. If I don't use per user credientials - I think google apps will change the sender address to a fixed sender address. It seems that you don't agree with me here? Also - part of the rationale of sending via smtp.google.com with user credentials - is to ensure that the google apps users sent email folders are maintained correctly. Perhaps you are describing an alternative method for google apps smtp which I am unaware of. If so - can you point me to a description of this alternative option? The implementation I am familiar with has users submitting mail directly via Gmail (Webmail or SMTP, their choice). Mail from Google hosted users, even to other Google hosted users, flows through a corporate (non-Google) relay and only then to its destination. Mail to Google hosted users is only accepted from the corporate relay, and the MX records are not handled by Google. In other words Gmail is just a large alternative mailstore, but is not the edge SMTP service. It is not clear what your use-case is, perhaps you should describe it in more detail. I probably misunderstood what you are trying to achieve in my initial reply. -- Viktor.
Re: reject_unknown_client_hostname light?
pf at alt-ctrl-del.org put forth on 9/7/2010 11:02 PM: Am I missing something obvious? With many ISPs providing generic PTR, reject_unknown_reverse_client_hostname is too gentle. I'd really like to implement reject_unknown_client_hostname, but I've seen too many cases where address-name mapping = exists, the name-address mapping = exists, BUT the name-address mapping is in the same /24 - but off by a couple of IPs. Is there a test that I'm missing out on that is simply address-name mapping = exists, and name-any address mapping = exists? Or a chain of tests that can accomplish the same thing? The battle against spam coming from generic rDNS clients isn't new. I suggest you try this combo for a while. The pcre file rejects generic rDNS patterns covering a large section of ISPs in the US, Canada, Europe, and elsewhere. I've been using it for quite a while with good results, as have a few others on this list. I'll let them speak for their results with it, if they so choose. This pcre doesn't cover all the ISPs on the planet, so you may want/need to add to it over time. ... reject_unknown_reverse_client_hostname ... check_client_access pcre:/etc/postfix/fqrdns.pcre ... http://www.hardwarefreak.com/fqrdns.pcre -- Stan
Reading mail messages from local files
Hello all, I'm setting up a mail server that needs to read messages that are created on the disk as individual files. This is an example file: From: Test 123 t...@localhost To: Diego Lima t...@domain.com Content-Type: text/plain; charset=iso-8859-1 MIME-Type: text/plain MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Test Message . What is the best way to read those messages? I considered creating a shellscript that checks the directory for new files and then sends them using sendmail -t, but that isn't really good performance-wise. Is there any way I can inject the messages into postfix queue? Thanks! -- Diego Lima
Re: Reading mail messages from local files
Diego Lima li...@diegolima.org writes: Hello all, I'm setting up a mail server that needs to read messages that are created on the disk as individual files. This is an example file: From: Test 123 t...@localhost To: Diego Lima t...@domain.com Content-Type: text/plain; charset=iso-8859-1 MIME-Type: text/plain MIME-Version: 1.0 Content-Transfer-Encoding: 8bit What is the best way to read those messages? I considered creating a shellscript that checks the directory for new files and then sends them using sendmail -t, but that isn't really good performance-wise. Is there any way I can inject the messages into postfix queue? man mailx(1) -Dieter -- Dieter Klünter | Systemberatung sip: 7770...@sipgate.de http://www.dpunkt.de/buecher/2104.html GPG Key ID:8EF7B6C6
Re: Reading mail messages from local files
Hi Dieter, I think I might have badly expressed myself :) The files on the disk still need to be sent to the addresses in the To field. They have simply been generated using an external program that can't talk smtp directly with my postfix server and needs it to pick up and deliever the messages. 2010/9/8 Dieter Kluenter die...@dkluenter.de: Diego Lima li...@diegolima.org writes: Hello all, I'm setting up a mail server that needs to read messages that are created on the disk as individual files. This is an example file: From: Test 123 t...@localhost To: Diego Lima t...@domain.com Content-Type: text/plain; charset=iso-8859-1 MIME-Type: text/plain MIME-Version: 1.0 Content-Transfer-Encoding: 8bit What is the best way to read those messages? I considered creating a shellscript that checks the directory for new files and then sends them using sendmail -t, but that isn't really good performance-wise. Is there any way I can inject the messages into postfix queue? man mailx(1) -Dieter -- Dieter Klünter | Systemberatung sip: 7770...@sipgate.de http://www.dpunkt.de/buecher/2104.html GPG Key ID:8EF7B6C6 -- Diego Lima
Re: Reading mail messages from local files
Diego Lima put forth on 9/8/2010 2:46 PM: I considered creating a shellscript that checks the directory for new files and then sends them using sendmail -t, but that isn't really good performance-wise. Performance-wise? How many emails are you sending per minute? Unless you have others processes sucking up a lot of resources on this system, performance using this method shouldn't be an issue if we're talking about a sane number of outbound emails. -- Stan
Re: Reading mail messages from local files
On Wed, 2010-09-08 at 17:11 -0300, Diego Lima wrote: Hi Dieter, I think I might have badly expressed myself :) The files on the disk still need to be sent to the addresses in the To field. They have simply been generated using an external program that can't talk smtp directly with my postfix server and needs it to pick up and deliever the messages. I do a similar thing using perl and MIME::Lite. You'll have to parse the file to split the to/from and message body - then send. There are other perl modules related to sending email, MIME::Lite fit my needs. Vernon
Re: Reading mail messages from local files
Hi Stan, This is actually a server for a mail marketing company, so I can expect several thousands of messages per minute being sent from the system. That's why I was wondering if there was any way to get postfix to pick up the messages automatically (the less programs/scripts in the way, the better) 2010/9/8 Stan Hoeppner s...@hardwarefreak.com: Diego Lima put forth on 9/8/2010 2:46 PM: I considered creating a shellscript that checks the directory for new files and then sends them using sendmail -t, but that isn't really good performance-wise. Performance-wise? How many emails are you sending per minute? Unless you have others processes sucking up a lot of resources on this system, performance using this method shouldn't be an issue if we're talking about a sane number of outbound emails. -- Stan -- Diego Lima
Re: Reading mail messages from local files
* Diego Lima li...@diegolima.org: Hi Stan, This is actually a server for a mail marketing company, so I can expect several thousands of messages per minute being sent from the system. That's why I was wondering if there was any way to get postfix to pick up the messages automatically (the less programs/scripts in the way, the better) Picking up sounds like sendmail command to me. This would be the slowest way to inject mails into the Postfix mail server. Make it the job of the mail marketing application to implement a SMTP client that hands messages over to Postfix. p...@rick 2010/9/8 Stan Hoeppner s...@hardwarefreak.com: Diego Lima put forth on 9/8/2010 2:46 PM: I considered creating a shellscript that checks the directory for new files and then sends them using sendmail -t, but that isn't really good performance-wise. Performance-wise? How many emails are you sending per minute? Unless you have others processes sucking up a lot of resources on this system, performance using this method shouldn't be an issue if we're talking about a sane number of outbound emails. -- Stan -- Diego Lima -- All technical questions asked privately will be automatically answered on the list and archived for public access unless privacy is explicitely required and justified. saslfinger (debugging SMTP AUTH): http://postfix.state-of-mind.de/patrick.koetter/saslfinger/
Remove Received: headers
Hi all. We have a local postfix server that relays to another postfix server via VPN and I would like to remove the Received: headers so that only the last one is included in the email message. How do I proceed to do that? Do I need to set up altermime? Thanks -JK
Re: Remove Received: headers
* Jack Knowlton jknowl...@vp44.com: Hi all. We have a local postfix server that relays to another postfix server via VPN and I would like to remove the Received: headers so that only the last one is included in the email message. How do I proceed to do that? Do I need to set up altermime? $ man header_checks | less +/IGNORE -- All technical questions asked privately will be automatically answered on the list and archived for public access unless privacy is explicitely required and justified. saslfinger (debugging SMTP AUTH): http://postfix.state-of-mind.de/patrick.koetter/saslfinger/
Re: Reading mail messages from local files
On Wed, Sep 08, 2010 at 05:33:40PM -0300, Diego Lima wrote: This is actually a server for a mail marketing company, so I can expect several thousands of messages per minute being sent from the system. A company in the business of sending email is expected to use tooling sufficiently sophisticated to talk SMTP. Dumping message files to disk is rather naive... That's why I was wondering if there was any way to get postfix to pick up the messages automatically (the less programs/scripts in the way, the better) You need a parser that runs a few parallel jobs to scan the queue and submit the queued files via SMTP (with an appropriate contention management scheme or a global scheduler). The Postfix pickup(8) service is single-threaded, and may not keep up with several thousand messages a minute depending on how may 'several' is. If your disk latency is low enough and you avoid high-latency lookup tables ..., you may be able to push pickup(8) over 50 msgs/sec or so. To submit a file that looks like an RFC822 message, you just need to invoke: sendmail -f 'envelope-sender' -t file and handle non-zero exit codes gracefully. Don't use the -i option if the files use . as an end-of-message marker, and double-up leading dots on non-terminal lines. Otherwise use the -i option. -- Viktor.
Re: set envelope sender = sasl authenticated user ?
Le 07/09/2010 23:36, Jan-Frode Myklebust a écrit : On Tue, Sep 07, 2010 at 08:20:36PM +0200, Jeroen Geilman wrote: On 09/07/2010 06:57 PM, mouss wrote: OP is an ISP providing outbound relay to residential users. his problem is not easy to solve. Thanks for understanding. I´ve gotten information off list that gmail are setting the sender to the gmail authenticated user when sending from non-gmail addresses. So I´m at least not alone in thinking this should be a good solution. forget about gmail. you have a serious problem, and the solution isnt' gmail. gmail is a public provider, you are not. you have a different problem, and a serious one. most of us want you to block outound spam (by blocking port 25 and filtering outbound mail). not easy... do not try to rewrite mail headers. this is the wrong approach. headers are specified by that who writes the message. instead, detect abuse/spam by counting the number of messages sent from a given IP and block the IP if it exceeds its quota. (block all its communication, not just smtp). Also I believe you have to pre-register any address you want to send from trough gmail, which sounds like we could use postfix´ smtpd_sender_login_maps to implement a similar solution. Residential users don't often have their own mail servers/domains. Limiting these to sending with their true ISP address is fairly common. (And just about the only sane way to implement this kind of relay) In Norway all/most ISPs are forcing their residential users trough the ISP`s smarthosts. Outgoing port 25/tcp is blocked. So users are not allowed to run their own mailservers. I can´t justify also requiring all users to use the ISP´s mail addresses. -jf
Re: set envelope sender = sasl authenticated user ?
Le 08/09/2010 10:44, Jan-Frode Myklebust a écrit : On Tue, Sep 07, 2010 at 06:38:15PM -0500, Noel Jones wrote: If you have customers sending large amounts of abusive mail, seems as if there would be better ways to deal with that eg. sender quotas, monitoring of undeliverable mail, inbound spam/virus scanning, etc. But I'm not an ISP; I can fire anyone who abuses the mail system. We do sender quotas, some monitoring of undeliverable mail, in and outbound spam/virus scanning and more, but this doesn't catch all. Users that are clearly abusing the system (read: has malware installed) gets blocked automatically or manually, but there's a time window where they will be able to send out junk, and when you have enough customers -- someone will always have the latest and greatest malware installed and we woun't catch it immediately. There are at least two different kind of users: - victims whose PCs are owned. here, network quotas, errors detection, ... will help you know. now what can you do? - spammers. you ought to detect them. but they can get back with different names, ... and I still fail to understand how controlling your customers envelope sender will help with backscatterer.org. It will make sure that when viruses/malware on the customers computer is sending out spam from fake addresses, the bounces goes back to the customer with the infected computer -- instead of to whomever the malware was pretending to send from. nah. this is useless. spam and viruses should get discarded. they have no reason getting on the wire. -jf
Re: Reading mail messages from local files
Diego Lima put forth on 9/8/2010 3:33 PM: Hi Stan, This is actually a server for a mail marketing company, so I can expect several thousands of messages per minute being sent from the system. That's why I was wondering if there was any way to get postfix to pick up the messages automatically (the less programs/scripts in the way, the better) email marketing in 99.99% of cases = spamming Please don't enable spammers. The spam problem is bad enough as it is. If you insist on doing so, please use qmail or another MTA to send the spam. Also, please be kind enough to inform this list which IP addresses these emails will be sent from so we can proactively block those IPs. Thanks for being a responsible SA. -- Stan
