Re: providing queue id for the clients
Zsombor B: > Hi, > > > > Please provide evidence. > > This is the point. :) > > External client sent us a mail we accepted with queue id "A". > I have asked them to look for this "A" in their logs. > I was told they can't find it in their logs. Postfix also logs a Message-Id, which should be the same in their logs. Feb 9 02:15:17 spike postfix/smtpd[1248]: 4DZZ0j1z3LzJrNw: client=english-breakfast.cloud9.net[2604:8d00:0:1::7] Feb 9 02:15:17 spike postfix/cleanup[1252]: 4DZZ0j1z3LzJrNw: message-id=<20210209071515.7d25340...@mail.example.com> Wietse > > Zsombor > > > > Id?zet (Wietse Venema ): > > > Zsombor B: > >> It turned out during an investigation that our postfix servers don't > >> provide a queue id for the external clients when accepting a new email. > > > > Please provide evidence. > > > > Postfix SMTP client logging: > > ... status=sent (250 2.0.0 Ok: queued as AA92365E6F) > > > > Wietse > > >
Re: providing queue id for the clients
Hi, Please provide evidence. This is the point. :) External client sent us a mail we accepted with queue id "A". I have asked them to look for this "A" in their logs. I was told they can't find it in their logs. Zsombor Idézet (Wietse Venema ): Zsombor B: It turned out during an investigation that our postfix servers don't provide a queue id for the external clients when accepting a new email. Please provide evidence. Postfix SMTP client logging: ... status=sent (250 2.0.0 Ok: queued as AA92365E6F) Wietse
Re: providing queue id for the clients
On 09 Feb 2021, at 05:45, Wietse Venema wrote:
> Zsombor B:
>> It turned out during an investigation that our postfix servers don't
>> provide a queue id for the external clients when accepting a new email.
>
> Please provide evidence.
>
> Postfix SMTP client logging:
> ... status=sent (250 2.0.0 Ok: queued as AA92365E6F)
The only thing that I see is that the queue id is a different form for mail
user on the server versus mails to external addresses.
# grep "status=sent" /var/log/mail.log | \
awk '{for(i=13;i<=NF;i++) printf $i" "; print ""}
(250 2.0.0 yIXGF1V9ImDhVgAAIdGjjQ Saved)
(250 2.0.0 Ok: queued as B1AA63AB070)
Perhaps the different formatting from "enable_long_queue_ids = yes" is
confusing the OP?
--
If I were you boys, I wouldn't talk or even think about women.
'T'ain't good for your health.
Re: providing queue id for the clients
Zsombor B: > It turned out during an investigation that our postfix servers don't > provide a queue id for the external clients when accepting a new email. Please provide evidence. Postfix SMTP client logging: ... status=sent (250 2.0.0 Ok: queued as AA92365E6F) Wietse
providing queue id for the clients
Hi, It turned out during an investigation that our postfix servers don't provide a queue id for the external clients when accepting a new email. However the very same servers do provide queue id for internal mail servers. Is there a specific configuration option to provide the queue id under any circumstances? Thank you, Zsombor
Re: Queue ID availability for milters on multi-message connections/sessions?
Wietse Venema wrote: Below are the SMTP commands/responses, and the test-milter output showing that the second "DATA" event is reported with the correct queue ID. OK, thanks! I'll take it up further with the milter authors. -kgd
Re: Queue ID availability for milters on multi-message connections/sessions?
Below are the SMTP commands/responses, and the test-milter output
showing that the second "DATA" event is reported with the correct
queue ID.
Wietse
$ telnet 127.0.0.1 smtp
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220-wzv.porcupine.org ESMTP Postfix
220 wzv.porcupine.org ESMTP Postfix
ehlo wzv.porcupine.org
250-wzv.porcupine.org
250-PIPELINING
250-SIZE 1024
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250 SMTPUTF8
mail from:<>
250 2.1.0 Ok
rcpt to:<wietse@localhost>
250 2.1.5 Ok
data
354 End data with .
blah
.
250 2.0.0 Ok: queued as 8E063A009F
mail from:<>
250 2.1.0 Ok
rcpt to:<wietse@localhost>
250 2.1.5 Ok
data
354 End data with .
blah
.
250 2.0.0 Ok: queued as 2751DA009E
quit
221 2.0.0 Bye
Connection closed by foreign host.
$ exit
exit
$ ./test-milter -d 1 -p inet:@127.0.0.1
test_connect localhost AF_INET (127.0.0.1:44670)
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
test_reply 0
test_helo "wzv.porcupine.org"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
test_reply 0
test_mail "<>"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
macro: {mail_addr}=""
macro: {mail_host}="wzv.porcupine.org"
macro: {mail_mailer}="local"
test_reply 0
test_rcpt "<wietse@localhost>"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
macro: {mail_addr}=""
macro: {mail_host}="wzv.porcupine.org"
macro: {mail_mailer}="local"
macro: {rcpt_addr}="wietse@localhost"
macro: {rcpt_host}="wzv.porcupine.org"
macro: {rcpt_mailer}="local"
test_reply 0
test_data
macro: i="8E063A009F"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
macro: {mail_addr}=""
macro: {mail_host}="wzv.porcupine.org"
macro: {mail_mailer}="local"
macro: {rcpt_addr}="wietse@localhost"
macro: {rcpt_host}="wzv.porcupine.org"
macro: {rcpt_mailer}="local"
test_reply 0
test_header "Message-Id" "<20170329233029.8e063a0...@wzv.porcupine.org>"
macro: i="8E063A009F"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
macro: {mail_addr}=""
macro: {mail_host}="wzv.porcupine.org"
macro: {mail_mailer}="local"
macro: {rcpt_addr}="wietse@localhost"
macro: {rcpt_host}="wzv.porcupine.org"
macro: {rcpt_mailer}="local"
test_reply 0
test_header "Date" "Wed, 29 Mar 2017 19:30:20 -0400 (EDT)"
macro: i="8E063A009F"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
macro: {mail_addr}=""
macro: {mail_host}="wzv.porcupine.org"
macro: {mail_mailer}="local"
macro: {rcpt_addr}="wietse@localhost"
macro: {rcpt_host}="wzv.porcupine.org"
macro: {rcpt_mailer}="local"
test_reply 0
test_header "From" "MAILER-DAEMON"
macro: i="8E063A009F"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
macro: {mail_addr}=""
macro: {mail_host}="wzv.porcupine.org"
macro: {mail_mailer}="local"
macro: {rcpt_addr}="wietse@localhost"
macro: {rcpt_host}="wzv.porcupine.org"
macro: {rcpt_mailer}="local"
test_reply 0
test_eoh
macro: i="8E063A009F"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1"
macro: {daemon_name}="wzv.porcupine.org"
macro: {mail_addr}=""
macro: {mail_host}="wzv.porcupine.org"
macro: {mail_mailer}="local"
macro: {rcpt_addr}="wietse@localhost"
macro: {rcpt_host}="wzv.porcupine.org"
macro: {rcpt_mailer}="local"
test_reply 0
test_body 6 bytes
macro: i="8E063A009F"
macro: j="wzv.porcupine.org"
macro: v="Postfix 3.3-20170212"
macro: {daemon_addr}="127.0.0.1&qu
Re: Queue ID availability for milters on multi-message connections/sessions?
Kris Deugau:
> Mar 29 16:35:14 jessie64 postfix/smtpd[17537]: connect from
> localhost[127.0.0.1]
> Mar 29 16:35:27 jessie64 postfix/smtpd[17537]: 26F5E428A4:
> client=localhost[127.0.0.1]
> Mar 29 16:36:02 jessie64 postfix/cleanup[17556]: 26F5E428A4:
> message-id=<f...@bar.com>
> Mar 29 16:36:03 jessie64 mimedefang.pl[17552]: 26F5E428A4:
> clientip=127.0.0.1 smtpauth= from=kdeu...@deepnet.cx to=f...@example.com
> messageid="<f...@bar.com>" queueid=26F5E428A4"
> Mar 29 16:36:03 jessie64 postfix/qmgr[17527]: 26F5E428A4:
> from=<kdeu...@deepnet.cx>, size=324, nrcpt=1 (queue active)
> Mar 29 16:36:03 jessie64 postfix/local[17557]: 26F5E428A4:
> to=<f...@jessie64.pem-lan>, orig_to=<f...@example.com>, relay=local,
> delay=42, delays=42/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to
> command: procmail -a "$EXTENSION")
> Mar 29 16:36:03 jessie64 postfix/qmgr[17527]: 26F5E428A4: removed
> Mar 29 16:36:17 jessie64 postfix/smtpd[17537]: 55ADB428A4:
> client=localhost[127.0.0.1]
> Mar 29 16:36:36 jessie64 postfix/cleanup[17556]: 55ADB428A4:
> message-id=<f...@bar.com>
> Mar 29 16:36:36 jessie64 mimedefang.pl[17552]: clientip=127.0.0.1
> smtpauth= from=kdeu...@deepnet.cx to=b...@example.com
> messageid="<f...@bar.com>" queueid=NOQUEUE"
> Mar 29 16:36:36 jessie64 postfix/qmgr[17527]: 55ADB428A4:
> from=<kdeu...@deepnet.cx>, size=324, nrcpt=1 (queue active)
> Mar 29 16:36:36 jessie64 postfix/local[17557]: 55ADB428A4:
> to=<f...@jessie64.pem-lan>, orig_to=<b...@example.com>, relay=local,
> delay=27, delays=27/0/0/0.03, dsn=2.0.0, status=sent (delivered to
> command: procmail -a "$EXTENSION")
> Mar 29 16:36:36 jessie64 postfix/qmgr[17527]: 55ADB428A4: removed
> Mar 29 16:36:38 jessie64 postfix/smtpd[17537]: disconnect from
> localhost[127.0.0.1] helo=1 mail=2 rcpt=2 data=2 quit=1 commands=8
I can't reproduce this. The Milter sends the queue ID as soon as
it is available (when a message has one recipient, that is with the
DATA command), with each message header, with end of headers, with
body, and with end-of-message. Same behavior with the first and
second message in an SMTP session. I observe this with the Postfix
test-milter program, and by looking at a tcpdump recording of the
traffic between Postfix and milter.
This is with default Postfix settings for macros (no milter-dependent
settings):
milter_mail_macros = i {auth_type} {auth_authen} {auth_author} {mail_addr}
{mail_host} {mail_mailer}
milter_rcpt_macros = i {rcpt_addr} {rcpt_host} {rcpt_mailer}
milter_data_macros = i
milter_end_of_header_macros = i
milter_end_of_data_macros = i
With these settings Postfix dies not send a macro unless it actually
has a value. Postfix never uses NOQUEUE as a queue ID and never
sends NOQUEUE to the milter (confirmed by test-milter program and
tcpdump recording). Thus, your milter is making this up.
Wietse
Re: Queue ID availability for milters on multi-message connections/sessions?
Wietse Venema wrote: Kris Deugau: I came across a bit of an information-passing glitch on a system that uses a milter (MIMEDefang) to glue together complex filter policies. MIMEDefang is configured to log sender, first recipient, Message-ID (if any), and the queue ID, along with some filter result data, for each message. This works just fine for messages sent on their own connection. However, if a remote system sends more than one message during a connection/session, the queue IDs of the second and further messages are not passed to/retrieved by the milter; instead they're logged as "NOQUEUE". I've confirmed this works as expected with sendmail, but not with any version of Postfix up through the current 3.3 snapshot. Logged as NOQUEUE by Postfix? Milter? Something else? At what protocol stage? Envelope? End-of-body? Something else? Actual logging would clarify a lot. I don't haver the time to drop what I'm doing and spend hours to reverse engineer your conditions. Sorry, guess I've been prodding this on and off for too long and forgot some of that wasn't as obvious as to me. The literal 'NOQUEUE' is a placeholder MIMEDefang initializes its local reference copy of the i macro to. smfi_getsymval(ctx, "i") is called in five places including end-of-message in the C that talks directly to libmilter. The log action I've added that creates the mimedefang.pl log entries below is in a section run at end-of-message. Here's the full mail log from a test system: Mar 29 16:35:14 jessie64 postfix/smtpd[17537]: connect from localhost[127.0.0.1] Mar 29 16:35:27 jessie64 postfix/smtpd[17537]: 26F5E428A4: client=localhost[127.0.0.1] Mar 29 16:36:02 jessie64 postfix/cleanup[17556]: 26F5E428A4: message-id=<f...@bar.com> Mar 29 16:36:03 jessie64 mimedefang.pl[17552]: 26F5E428A4: clientip=127.0.0.1 smtpauth= from=kdeu...@deepnet.cx to=f...@example.com messageid="<f...@bar.com>" queueid=26F5E428A4" Mar 29 16:36:03 jessie64 postfix/qmgr[17527]: 26F5E428A4: from=<kdeu...@deepnet.cx>, size=324, nrcpt=1 (queue active) Mar 29 16:36:03 jessie64 postfix/local[17557]: 26F5E428A4: to=<f...@jessie64.pem-lan>, orig_to=<f...@example.com>, relay=local, delay=42, delays=42/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION") Mar 29 16:36:03 jessie64 postfix/qmgr[17527]: 26F5E428A4: removed Mar 29 16:36:17 jessie64 postfix/smtpd[17537]: 55ADB428A4: client=localhost[127.0.0.1] Mar 29 16:36:36 jessie64 postfix/cleanup[17556]: 55ADB428A4: message-id=<f...@bar.com> Mar 29 16:36:36 jessie64 mimedefang.pl[17552]: clientip=127.0.0.1 smtpauth= from=kdeu...@deepnet.cx to=b...@example.com messageid="<f...@bar.com>" queueid=NOQUEUE" Mar 29 16:36:36 jessie64 postfix/qmgr[17527]: 55ADB428A4: from=<kdeu...@deepnet.cx>, size=324, nrcpt=1 (queue active) Mar 29 16:36:36 jessie64 postfix/local[17557]: 55ADB428A4: to=<f...@jessie64.pem-lan>, orig_to=<b...@example.com>, relay=local, delay=27, delays=27/0/0/0.03, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION") Mar 29 16:36:36 jessie64 postfix/qmgr[17527]: 55ADB428A4: removed Mar 29 16:36:38 jessie64 postfix/smtpd[17537]: disconnect from localhost[127.0.0.1] helo=1 mail=2 rcpt=2 data=2 quit=1 commands=8 and the SMTP transcript: # telnet localhost 25 Trying ::1... Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 jessie64.pem-lan ESMTP Postfix helo local 250 jessie64.pem-lan mail from:kdeu...@deepnet.cx 250 2.1.0 Ok rcpt to:f...@example.com 250 2.1.5 Ok data 354 End data with . Subject: test Message-ID: <f...@bar.com> test . 250 2.0.0 Ok: queued as 26F5E428A4 mail from:kdeu...@deepnet.cx 250 2.1.0 Ok rcpt to:b...@example.com 250 2.1.5 Ok data 354 End data with . Subject: test Message-ID: <f...@bar.com> test . 250 2.0.0 Ok: queued as 55ADB428A4 quit 221 2.0.0 Bye Connection closed by foreign host. # For completeness: # sbin/postconf -n alias_database = hash:/opt/pfcustom/etc/aliases alias_maps = hash:/opt/pfcustom/etc/aliases append_dot_mydomain = no biff = no compatibility_level = 2 default_transport = error inet_protocols = ipv4 mailbox_command = procmail -a "$EXTENSION" mailbox_size_limit = 0 mydestination = jessie64.pem-lan, localhost.pem-lan, localhost, jessie64.deepnet.cx, jessie64, example.com myhostname = jessie64.pem-lan mynetworks = 127.0.0.0/8 [:::127.0.0.0]/104 [::1]/128 192.168.2.0/24 myorigin = jessie64.pem-lan readme_directory = no recipient_delimiter = + relay_transport = error relayhost = smtpd_milters = unix:/var/spool/MIMEDefang/mimedefang.sock smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination virtual_alias_maps = texthash:/opt/pfcustom/etc/virtual_alias # -kgd
Re: Queue ID availability for milters on multi-message connections/sessions?
Kris Deugau: > I came across a bit of an information-passing glitch on a system that > uses a milter (MIMEDefang) to glue together complex filter policies. > > MIMEDefang is configured to log sender, first recipient, Message-ID (if > any), and the queue ID, along with some filter result data, for each > message. > > This works just fine for messages sent on their own connection. > > However, if a remote system sends more than one message during a > connection/session, the queue IDs of the second and further messages are > not passed to/retrieved by the milter; instead they're logged as > "NOQUEUE". I've confirmed this works as expected with sendmail, but not > with any version of Postfix up through the current 3.3 snapshot. Logged as NOQUEUE by Postfix? Milter? Something else? At what protocol stage? Envelope? End-of-body? Something else? Actual logging would clarify a lot. I don't haver the time to drop what I'm doing and spend hours to reverse engineer your conditions. Wietse
Queue ID availability for milters on multi-message connections/sessions?
I came across a bit of an information-passing glitch on a system that uses a milter (MIMEDefang) to glue together complex filter policies. MIMEDefang is configured to log sender, first recipient, Message-ID (if any), and the queue ID, along with some filter result data, for each message. This works just fine for messages sent on their own connection. However, if a remote system sends more than one message during a connection/session, the queue IDs of the second and further messages are not passed to/retrieved by the milter; instead they're logged as "NOQUEUE". I've confirmed this works as expected with sendmail, but not with any version of Postfix up through the current 3.3 snapshot. Does the milter need the fix near the bottom of http://www.postfix.org/MILTER_README.html copied somewhere else? Or is Postfix not (re?)setting this macro after the first message is done with? -kgd
Re: Queue id uniqueness
hyndavirap...@bel.co.in: I have enabled enable_long_queue_id = yes That is a non-existent parameter. Now my doubt is how long, queueids will be unique, for 150 mails/min mail flow? http://www.postfix.org/postconf.5.html#enable_long_queue_ids enable_long_queue_ids (default: no) Enable long, non-repeating, queue IDs (queue file names). The benefit of non-repeating names is simpler logfile analysis and easier queue migration (there is no need to run postsuper to change queue file names that don't match their message file inode number). Wietse
Re: Queue id uniqueness
Am 2015-01-22 10:26, schrieb hyndavirap...@bel.co.in: I have enabled enable_long_queue_id = yes Now my doubt is how long, queueids will be unique, for 150 mails/min mail flow? The id is build from the time and the file-id within the filesystem. So each queue_id should be locally unique. The long queueid will not repeat within a UNIX epoch. The default queueid will not repeat within a second. The code for generating the queue_id: https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/mail_queue.c#L393 The encoding is defined in: https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/mail_queue.h#L80 Markus -- Markus Benning, https://markusbenning.de/
Queue id uniqueness
Hi, I have a query regarding queue ids. In my mail system, 150 mails/min flow is there and I need to keep track of mails and DSN for atleast 24 hours. To achieve this functionality, I have modified code as follows. 1. I am logging all mails Queue ids in a text file.(Queueid represents Correspondiong mail) 2. If I am receiving DSN (DSN contains for which queueid, it is for), checking for which mail it is for and then updating corresponding queueid (mail) as delivered. 3. If I don't receive DSN for some time(I'm waiting for 10 hours), I need to send mail again(automatically). To achieve above, Queueids should be unique. I have enabled enable_long_queue_id = yes Now my doubt is how long, queueids will be unique, for 150 mails/min mail flow? or am I missing some thing crucial? or is there a simple way to do this? Is my approach correct? All your suggestions are appreciated. Thanking you in advance Regards, Hyndavi Every 3000 Sheets of paper costs us a tree.. Save trees... Conserve Trees. Don't print this email or any Files unless you really need to Confidentiality Notice The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify the sender at Bharat Electronics or supp...@bel.co.in immediately and destroy all copies of this message and any attachments.
Re: Correlate client IP address with queue ID
Rolf E. Sonneveld: Apr 23 20:26:38 helium postfix-cust1/smtpd[9220]: 3ZwCmG272nz1L8Zd: client=D57E1702.static.ziggozakelijk.nl[213.126.23.2] The above logging gives you the link between client and queue ID. client = D57E1702.static.ziggozakelijk.nl[213.126.23.2] queue ID = 3ZwCmG272nz1L8Zd From here on, the smtp daemon, cleanup daemon, queue manager, and delivery agents will show the same queue ID in status reports. You're using long queue IDs which never repeat (unless someone resets the clock. don't). In addition, the logging shows the process that logs the information: process name = smtpd process ID = 9220 Each process handles one or more deliveries during its life time, so the process ID (9220 above) is not unique for this particular delivery. Wietse
Re: Correlate client IP address with queue ID
On 04/23/2013 10:14 PM, Wietse Venema wrote: Rolf E. Sonneveld: Apr 23 20:26:38 helium postfix-cust1/smtpd[9220]: 3ZwCmG272nz1L8Zd: client=D57E1702.static.ziggozakelijk.nl[213.126.23.2] The above logging gives you the link between client and queue ID. client = D57E1702.static.ziggozakelijk.nl[213.126.23.2] queue ID = 3ZwCmG272nz1L8Zd From here on, the smtp daemon, cleanup daemon, queue manager, and delivery agents will show the same queue ID in status reports. You're using long queue IDs which never repeat (unless someone resets the clock. don't). In addition, the logging shows the process that logs the information: process name = smtpd process ID = 9220 Each process handles one or more deliveries during its life time, so the process ID (9220 above) is not unique for this particular delivery. thanks Wietse and /dev/rob0, I must have been blind not to see this! Apologies for having taken your time, but thanks for the continuing support! /rolf
Re: AW: How to change queue id?
On 10/03/2012 11:30 PM, Steffen Schebesta wrote: Thanks for all the insightful answers. So, I actually use the long_queue_ids options and I save the queue_ids to a database to later compare them to the queue_ids found in the mail log to parse and mark the bounces. The problem - and thus the source for my question - is that this always means a string compare in the database search which is aweful for performance. Not if you index the queue-ID properly. If the database is central to your bounce handling process, you need this anyway. -- J.
RE: AW: AW: How to change queue id?
Hello Wietse, ok, here is my problem in detail: I pass mails to Postfix through smtpd. Postfix sends them out and returns the queue_id. I save the queue_id to a database (UPDATE table SET mail_id = '...' WHERE id = ...). in a table structured like this: ID (INT, auto increment, indexed) - unique id for internal purpose mail_id (VARCHAR) - the queue_id from Postfix html (TEXT) - the html part of the message text (TEXT) - the text part of the message recipient (VARCHAR) - the email address of the recipient status (VARCHAR) - the status of the email (can be sent or bounced, when the message bounced, it saves the dsn and reason as well) I parse the mail.log for bounces every hour. When I find a bounce in the log I look up the mail_id in the db table and set the status to bounced (UPDATE table SET status = bounced WHERE mail_id = '...'). So for every bounce I need two database queries: 1. The update to save the queue_id 2. The update to set the status to bounced This is not efficient and performs badly when the table becomes bigger because the second query takes a long time. Reasons: 1. Text compare in WHERE clause 2. no index on mail_id I cannot set an index on the mail_id since that would slow down the inserting process to much (believe me, I have tried that option). So that was the problem. And now here is what I thought would be an adequate solution: Since I already have a unique id in the table I could use that instead of the mail_id. It would eliminate query no. 1 and speed up query no. 2 by an order-of-magnitude since the WHERE clause would only include one indexed column (id). So I thought I could find a way with Postfix to pass my internal table id to the mail.log when a bounce is logged and then parse it after. That's why I tried changing the queue_id to my internal id and when that didn't work tried to set the MAIL FROM extension to the internal id but it is not logged in the same line making the parsing unreliable (the mail from and the bounce notification are connected through the queue_id but are logged on different lines that could end up in different files when parsed every hour). I hope the explanation of the problem helped to find a solution, Best, Steffen -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Wietse Venema Sent: Sonntag, 7. Oktober 2012 13:59 To: Postfix users Subject: Re: AW: AW: How to change queue id? Steffen Schebesta: Hello Witse, I really don't want to get on your nerves but as far as I understand using the standardized bounce messages from Postfix Start explaining the problem. Stop talking about what you think are solutions. Stop wasting everyone's time on this mailing list. You have an application that generates email. You have an MTA that tries to deliver that email. Now what is the problem? Removing obsolete addresses from the list? Stop talking about solutions like changing the queue ID or changing the Postfix logging. Wietse
Re: How to change queue id?
On Mon, Oct 08, 2012 at 10:25:20AM +0200, Steffen Schebesta wrote: ok, here is my problem in detail: I pass mails to Postfix through smtpd. Stop here. What are you trying to do? What sort of mail? This sounds like bulk email. Is it? Postfix sends them out and returns the queue_id. I save the queue_id to a database (UPDATE table SET mail_id = '...' WHERE id = ...). in a table structured like this: ID (INT, auto increment, indexed) - unique id for internal purpose mail_id (VARCHAR) - the queue_id from Postfix html (TEXT) - the html part of the message text (TEXT) - the text part of the message recipient (VARCHAR) - the email address of the recipient status (VARCHAR) - the status of the email (can be sent or bounced, when the message bounced, it saves the dsn and reason as well) I parse the mail.log for bounces every hour. When I find a bounce in the log I look up the mail_id in the db table and set the status to bounced (UPDATE table SET status = bounced WHERE mail_id = '...'). So for every bounce I need two database queries: 1. The update to save the queue_id 2. The update to set the status to bounced So the actual, real-world problem and goal: I am sending bulk email, and I need an automated means to track bounces for list maintenance (or list washing, as the case may be.) Am I close? If so, you are in luck: this is a solved problem. http://www.postfix.org/VERP_README.html Google the list archives for examples. VERP is your new search term. This is not efficient and performs badly when the table becomes bigger because the second query takes a long time. Reasons: 1. Text compare in WHERE clause 2. no index on mail_id I cannot set an index on the mail_id since that would slow down the inserting process to much (believe me, I have tried that option). So that was the problem. And now here is what I thought would be an adequate solution: Since I already have a unique id in the table I could use that instead of the mail_id. It would eliminate query no. 1 and speed up query no. 2 by an order-of-magnitude since the WHERE clause would only include one indexed column (id). So I thought I could find a way with Postfix to pass my internal table id to the mail.log when a bounce is logged and then parse it after. That's why I tried changing the queue_id to my internal id and when that didn't work tried to set the MAIL FROM extension to the internal id but it is not logged in the same line making the parsing unreliable (the mail from and the bounce notification are connected through the queue_id but are logged on different lines that could end up in different files when parsed every hour). I hope the explanation of the problem helped to find a solution, Why do you find it so difficult to describe the actual problem and goal? It took a whole, long discussion here, and we still have not gotten that! -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Wietse Venema Sent: Sonntag, 7. Oktober 2012 13:59 To: Postfix users Subject: Re: AW: AW: How to change queue id? Steffen Schebesta: Hello Witse, I really don't want to get on your nerves but as far as I understand using the standardized bounce messages from Postfix Start explaining the problem. Stop talking about what you think are solutions. Stop wasting everyone's time on this mailing list. You have an application that generates email. You have an MTA that tries to deliver that email. Now what is the problem? Removing obsolete addresses from the list? Stop talking about solutions like changing the queue ID or changing the Postfix logging. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if /dev/rob0 is in the Subject:
Re: AW: AW: How to change queue id?
On 10/8/2012 3:25 AM, Steffen Schebesta wrote: Hello Wietse, ok, here is my problem in detail: This isn't the answer to the question Wietse, and myself, asked you. You've failed to understand the nature of our question 3 times now. 10,000 ft view means you're in an airplane at altitude 10,000 feet looking down on the problem on the ground. All you can see is the outline, the overview, the big picture. You can't see any details. We want the big picture. Why did you create the database and what is its purpose? Possible answers: save the whales, win the lotto, etc Big picture. Not details. -- Stan I pass mails to Postfix through smtpd. Postfix sends them out and returns the queue_id. I save the queue_id to a database (UPDATE table SET mail_id = '...' WHERE id = ...). in a table structured like this: ID (INT, auto increment, indexed) - unique id for internal purpose mail_id (VARCHAR) - the queue_id from Postfix html (TEXT) - the html part of the message text (TEXT) - the text part of the message recipient (VARCHAR) - the email address of the recipient status (VARCHAR) - the status of the email (can be sent or bounced, when the message bounced, it saves the dsn and reason as well) I parse the mail.log for bounces every hour. When I find a bounce in the log I look up the mail_id in the db table and set the status to bounced (UPDATE table SET status = bounced WHERE mail_id = '...'). So for every bounce I need two database queries: 1. The update to save the queue_id 2. The update to set the status to bounced This is not efficient and performs badly when the table becomes bigger because the second query takes a long time. Reasons: 1. Text compare in WHERE clause 2. no index on mail_id I cannot set an index on the mail_id since that would slow down the inserting process to much (believe me, I have tried that option). So that was the problem. And now here is what I thought would be an adequate solution: Since I already have a unique id in the table I could use that instead of the mail_id. It would eliminate query no. 1 and speed up query no. 2 by an order-of-magnitude since the WHERE clause would only include one indexed column (id). So I thought I could find a way with Postfix to pass my internal table id to the mail.log when a bounce is logged and then parse it after. That's why I tried changing the queue_id to my internal id and when that didn't work tried to set the MAIL FROM extension to the internal id but it is not logged in the same line making the parsing unreliable (the mail from and the bounce notification are connected through the queue_id but are logged on different lines that could end up in different files when parsed every hour). I hope the explanation of the problem helped to find a solution, Best, Steffen -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Wietse Venema Sent: Sonntag, 7. Oktober 2012 13:59 To: Postfix users Subject: Re: AW: AW: How to change queue id? Steffen Schebesta: Hello Witse, I really don't want to get on your nerves but as far as I understand using the standardized bounce messages from Postfix Start explaining the problem. Stop talking about what you think are solutions. Stop wasting everyone's time on this mailing list. You have an application that generates email. You have an MTA that tries to deliver that email. Now what is the problem? Removing obsolete addresses from the list? Stop talking about solutions like changing the queue ID or changing the Postfix logging. Wietse
Re: AW: AW: How to change queue id?
Steffen Schebesta: Hello Wietse, ok, here is my problem in detail: I pass mails to Postfix through smtpd. Postfix sends them out and returns the queue_id. ... I parse the mail.log for bounces every hour. When I find a bounce in the log You are re-inventing an old wheel: it's called list washing. This functionality is present in modern mailing list managers. One popular solution is to encode your unique ID in the SMTP MAIL FROM address, as an address extension. The unique ID includes the recipient, and perhaps information about the message such as a sequence number. Postfix reports all errors to the SMTP MAIL FROM address, as do many other RFC-compliant MTAs. This not only works for mail that bounces while Postfix delivers it (the status=bounced is logged on your server), this also works for mail that bounces AFTER Postfix delivers it (the status=bounced is logged on some REMOTE server). Wietse
Re: AW: AW: How to change queue id?
Steffen Schebesta: Hello Witse, I really don't want to get on your nerves but as far as I understand using the standardized bounce messages from Postfix Start explaining the problem. Stop talking about what you think are solutions. Stop wasting everyone's time on this mailing list. You have an application that generates email. You have an MTA that tries to deliver that email. Now what is the problem? Removing obsolete addresses from the list? Stop talking about solutions like changing the queue ID or changing the Postfix logging. Wietse
AW: AW: How to change queue id?
Using Wietse's first approach (adding a custom id to the MAIL FROM address as an extension) I have tried to output the sender's address in the same line of the mail.log as the bounce message. I believe I would need to change the global/log_adhoc.c (e.g. line 109 in 2.9.4) to do so but I cannot access the sender there. Is there a way to access the initial sender from the log_adhoc void? Maybe through the RECIPIENT object: RECIPIENT-QMGR_QUEUE-QMGR_ENTRY_LIST-QMGR_ENTRY-QMGR_MESSAGE-sender I know it sounds awefully complicated but I really need a custom id in the same line as the bounce message. Alternatively, if there is another location where I access and ouput the sender, the recipient, the status=bounced (reason) and the dsn that would also work. Any help is greatly appreciated. Best, Steffen -Ursprüngliche Nachricht- Von: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] Im Auftrag von Wietse Venema Gesendet: Donnerstag, 4. Oktober 2012 14:26 An: Postfix users Betreff: Re: AW: How to change queue id? Steffen Schebesta: If that doesn't work though then maybe I could work around this problem. I thought about adding the message-id to the bounce message but that probably Postfix has lots of options to identify a returned message. 1) You can encode the unique identifier in the envelope sender address (as an address extension). This works even when mail is returned by a remote MTA. 2) Your unique Message ID is already returned in the bounce message. See RFC 3462. This may or may not work for mail that is returned by a remote MTA. 3) You can specify your own unique ENVELOPE ID via SMTP (see RFC 3461) or via the Postfix sendmail -V option. This ENVELOPE ID is also returned in the bounce message. See RFC 3464. This works only if all MTAs in the forward path implement RFC 3461. I suggest that you study the many RFCs and non-RFC features that Postfix already implements, before tinkering with internals. But first, as Stan mentioned, you should describe your problem, instead of your proposed solutions. There are likely better solutions that already exist. I already mentioned three of them. Wietse
Re: How to change queue id?
Am 06.10.2012 18:20, schrieb Steffen Schebesta: Using Wietse's first approach (adding a custom id to the MAIL FROM address as an extension) I have tried to output the sender's address in the same line of the mail.log as the bounce message. I believe I would need to change the global/log_adhoc.c (e.g. line 109 in 2.9.4) to do so but I cannot access the sender there. Is there a way to access the initial sender from the log_adhoc void? Maybe through the RECIPIENT object: RECIPIENT-QMGR_QUEUE-QMGR_ENTRY_LIST-QMGR_ENTRY-QMGR_MESSAGE-sender I know it sounds awefully complicated but I really need a custom id in the same line as the bounce message this is NOT possible please try to understand how a QUEUE works you get ONE message with 10 RCPT some of them are local, some of them not the non-local have different destination servers some of them may not be reachable and tried later so you have on the postfix side many different log lines for the same message, and some of them may arrive 5 days later while some messages are still delivered sucessfully signature.asc Description: OpenPGP digital signature
Re: AW: AW: How to change queue id?
Steffen Schebesta: Using Wietse's first approach (adding a custom id to the MAIL FROM address as an extension) I have tried to output the sender's address in the same line of the mail.log as the bounce message. To this end, YOU specify the sender address AT MAIL SUBMISSION TIME, instead of tinkering with Postfix source which is not supported. Wietse
Re: AW: AW: How to change queue id?
Am 06.10.2012 um 23:22 schrieb Wietse Venema wie...@porcupine.org: Steffen Schebesta: Using Wietse's first approach (adding a custom id to the MAIL FROM address as an extension) I have tried to output the sender's address in the same line of the mail.log as the bounce message. To this end, YOU specify the sender address AT MAIL SUBMISSION TIME, instead of tinkering with Postfix source which is not supported. Steffen Schebesta: Sorry that I didn't make it clear. Of course I set the sender at submission time. All I want is to output the sender now when the bounce happens in the log together with the recipient address. This is unnecessary. And I repeat that tinkering with Postfix source code is not supported. Postfix returns a standardized bounce message with the old queue id AND the bounced recipient AND the sender-address-with-your-identier and the Message-ID and more. You don't need to parse logfiles. Wietse
Re: AW: How to change queue id?
On 10/3/2012 4:30 PM, Steffen Schebesta wrote: Thanks for all the insightful answers. So, I actually use the long_queue_ids options and I save the queue_ids to a database to later compare them to the queue_ids found in the mail log to parse and mark the bounces. The problem - and thus the source for my question - is that this always means a string compare in the database search which is aweful for performance. So my thought was, to always pass a unique Message-ID to postfix equal to the autoincrement ID in my database, and then set this Message-ID as the mail's queue_id. This way the lookup in the database would be much faster plus I wouldn't have to save the queue_id in the database in the first place (huge performance boost). In my case I can guarantee that the Message-ID is always passed and that it is unique, so that wouldn't be the problem. But if Postfix creates the queue file name before receiving the message content then it actually is a bigger problem than I thought, except if I could change the queue_id or append the id to the queue_id... If that doesn't work though then maybe I could work around this problem. I thought about adding the message-id to the bounce message but that probably won't work because the message-id would not be in Postfix anymore at the time the bounce happens. What's the end game here, what is it you're trying to accomplish? One sentence in plain English please, the 10,000 ft overview. E.g. I want to do X where X is 20 words or less. -- Stan Any other options that I could try? Really appreciate your help here! Best, Steffen -Ursprüngliche Nachricht- Von: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] Im Auftrag von Wietse Venema Gesendet: Mittwoch, 3. Oktober 2012 18:47 An: Postfix users Betreff: Re: How to change queue id? Steffen Schebesta: Hello everybody, I deliver mails to my Postfix through smtpd. Postfix then takes it and sends it out to the recipient. Now I'm trying to change the queue_id for each email in Postfix 2.9 source code so that it is equal to the Message-ID (it is unique, don't worry) that I set in the email header when passing the email to Postfix through smtpd. I've looked in the source code for hours but cannot find the first time queue_id is set. In smtpd.c that (in my understanding) handles incoming mail through smtpd the variable state-queue_id is already used and only read but not written. I would really appreciate any tips where I can find the initial setting and on how I can set state-queue_id to Message-ID Reasons not to use the message-id as the queue file name: - Different messages may contain the same Message-ID value. - Some messages don't contain a Message-ID. This field is not required by current RFC documents. - Chicken-and-egg problem: Postfix chooses the queue file name (and creates the file) before it receives the message content. I suggest that you set enable_long_queue_ids = yes. Long queue ID strings don't repeat as long as your clock moves forward. Wietse
Re: AW: How to change queue id?
Steffen Schebesta: If that doesn't work though then maybe I could work around this problem. I thought about adding the message-id to the bounce message but that probably Postfix has lots of options to identify a returned message. 1) You can encode the unique identifier in the envelope sender address (as an address extension). This works even when mail is returned by a remote MTA. 2) Your unique Message ID is already returned in the bounce message. See RFC 3462. This may or may not work for mail that is returned by a remote MTA. 3) You can specify your own unique ENVELOPE ID via SMTP (see RFC 3461) or via the Postfix sendmail -V option. This ENVELOPE ID is also returned in the bounce message. See RFC 3464. This works only if all MTAs in the forward path implement RFC 3461. I suggest that you study the many RFCs and non-RFC features that Postfix already implements, before tinkering with internals. But first, as Stan mentioned, you should describe your problem, instead of your proposed solutions. There are likely better solutions that already exist. I already mentioned three of them. Wietse
RE: AW: How to change queue id?
Hello Wietse, option 1 is actually a pretty good idea - hadn't thought of this. The disadvantage of handling the return mail is that not only bounces are returned but also automatic responders, etc. and thus I would have to parse the content of the email to find out if it is a real bounce. So I think parsing the mail.log is actually much easier... Options 2 and 3 are interesting ideas as well but - as you said yourself - aren't reliable enough. So in the end I still think a custom field in the mail.log when the bounce is reported would make most sense. Steffen -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Wietse Venema Sent: Donnerstag, 4. Oktober 2012 14:26 To: Postfix users Subject: Re: AW: How to change queue id? Steffen Schebesta: If that doesn't work though then maybe I could work around this problem. I thought about adding the message-id to the bounce message but that probably Postfix has lots of options to identify a returned message. 1) You can encode the unique identifier in the envelope sender address (as an address extension). This works even when mail is returned by a remote MTA. 2) Your unique Message ID is already returned in the bounce message. See RFC 3462. This may or may not work for mail that is returned by a remote MTA. 3) You can specify your own unique ENVELOPE ID via SMTP (see RFC 3461) or via the Postfix sendmail -V option. This ENVELOPE ID is also returned in the bounce message. See RFC 3464. This works only if all MTAs in the forward path implement RFC 3461. I suggest that you study the many RFCs and non-RFC features that Postfix already implements, before tinkering with internals. But first, as Stan mentioned, you should describe your problem, instead of your proposed solutions. There are likely better solutions that already exist. I already mentioned three of them. Wietse
How to change queue id?
Hello everybody, I deliver mails to my Postfix through smtpd. Postfix then takes it and sends it out to the recipient. Now I'm trying to change the queue_id for each email in Postfix 2.9 source code so that it is equal to the Message-ID (it is unique, don't worry) that I set in the email header when passing the email to Postfix through smtpd. I've looked in the source code for hours but cannot find the first time queue_id is set. In smtpd.c that (in my understanding) handles incoming mail through smtpd the variable state-queue_id is already used and only read but not written. I would really appreciate any tips where I can find the initial setting and on how I can set state-queue_id to Message-ID Thanks, Steffen
Re: How to change queue id?
Am 03.10.2012 16:53, schrieb Steffen Schebesta: I deliver mails to my Postfix through smtpd. Postfix then takes it and sends it out to the recipient. Now I'm trying to change the |queue_id| for each email in Postfix 2.9 source code so that it is equal to the |Message-ID| (it is unique, don't worry) that I set in the email header when passing the email to Postfix through smtpd. I've looked in the source code for hours but cannot find the first time |queue_id| is set do NOT touch it this was discussed several times signature.asc Description: OpenPGP digital signature
Re: How to change queue id?
On Wed, Oct 03, 2012 at 04:53:22PM +0200, Steffen Schebesta wrote: I deliver mails to my Postfix through smtpd. Postfix then takes it and sends it out to the recipient. One or more recipients. And it may receive one or more copies of the original message if it was sent to many recipients. Now I'm trying to change the queue_id for each email in Postfix 2.9 source code so that it is equal to the Message-ID (it is unique, don't worry) that You can't do that. There is no guarantee that each mail transaction (sender, recipients, message-content) has a unique message-id (part of the message-content) since multi-recipient messages may be split into multiple transactions in transit. I set in the email header when passing the email to Postfix through smtpd. I've looked in the source code for hours but cannot find the first time queue_id is set. You can stop looking. What you're trying to do is not viable. There are existing ways to message tracking via log parsing and success DSNs if you want to aumate tracking of outbound mail. Recent Postfix releases support non-recurring long queue-ids, they are not going to be equal to the message-id, but that's fine. -- Viktor.
Re: How to change queue id?
On 10/3/2012 9:53 AM, Steffen Schebesta wrote: Now I'm trying to change the |queue_id| for each email in Postfix 2.9 source code so that it is equal to the |Message-ID| (it is unique, don't worry) that I set in the email header when passing the email to Postfix through smtpd. Why? At any rate, this is not possible. The queueid is generated before the message is received; that cannot be changed without *extensive* surgery. -- Noel Jones
Re: How to change queue id?
Steffen Schebesta: Hello everybody, I deliver mails to my Postfix through smtpd. Postfix then takes it and sends it out to the recipient. Now I'm trying to change the queue_id for each email in Postfix 2.9 source code so that it is equal to the Message-ID (it is unique, don't worry) that I set in the email header when passing the email to Postfix through smtpd. I've looked in the source code for hours but cannot find the first time queue_id is set. In smtpd.c that (in my understanding) handles incoming mail through smtpd the variable state-queue_id is already used and only read but not written. I would really appreciate any tips where I can find the initial setting and on how I can set state-queue_id to Message-ID Reasons not to use the message-id as the queue file name: - Different messages may contain the same Message-ID value. - Some messages don't contain a Message-ID. This field is not required by current RFC documents. - Chicken-and-egg problem: Postfix chooses the queue file name (and creates the file) before it receives the message content. I suggest that you set enable_long_queue_ids = yes. Long queue ID strings don't repeat as long as your clock moves forward. Wietse
AW: How to change queue id?
Thanks for all the insightful answers. So, I actually use the long_queue_ids options and I save the queue_ids to a database to later compare them to the queue_ids found in the mail log to parse and mark the bounces. The problem - and thus the source for my question - is that this always means a string compare in the database search which is aweful for performance. So my thought was, to always pass a unique Message-ID to postfix equal to the autoincrement ID in my database, and then set this Message-ID as the mail's queue_id. This way the lookup in the database would be much faster plus I wouldn't have to save the queue_id in the database in the first place (huge performance boost). In my case I can guarantee that the Message-ID is always passed and that it is unique, so that wouldn't be the problem. But if Postfix creates the queue file name before receiving the message content then it actually is a bigger problem than I thought, except if I could change the queue_id or append the id to the queue_id... If that doesn't work though then maybe I could work around this problem. I thought about adding the message-id to the bounce message but that probably won't work because the message-id would not be in Postfix anymore at the time the bounce happens. Any other options that I could try? Really appreciate your help here! Best, Steffen -Ursprüngliche Nachricht- Von: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] Im Auftrag von Wietse Venema Gesendet: Mittwoch, 3. Oktober 2012 18:47 An: Postfix users Betreff: Re: How to change queue id? Steffen Schebesta: Hello everybody, I deliver mails to my Postfix through smtpd. Postfix then takes it and sends it out to the recipient. Now I'm trying to change the queue_id for each email in Postfix 2.9 source code so that it is equal to the Message-ID (it is unique, don't worry) that I set in the email header when passing the email to Postfix through smtpd. I've looked in the source code for hours but cannot find the first time queue_id is set. In smtpd.c that (in my understanding) handles incoming mail through smtpd the variable state-queue_id is already used and only read but not written. I would really appreciate any tips where I can find the initial setting and on how I can set state-queue_id to Message-ID Reasons not to use the message-id as the queue file name: - Different messages may contain the same Message-ID value. - Some messages don't contain a Message-ID. This field is not required by current RFC documents. - Chicken-and-egg problem: Postfix chooses the queue file name (and creates the file) before it receives the message content. I suggest that you set enable_long_queue_ids = yes. Long queue ID strings don't repeat as long as your clock moves forward. Wietse
Queue Id Collision
Hello, I use Postfix 2.5.5 on a linux debian box and i'm trying to make some statistics for sasl authenticated user... the goal is to show them how many connexions to the server they have been made, how many mails they have sent, and show them the detected from and to addresses... My search are based on Queue ID in the logs... finding a Queue Id, parse the log file to grep this Queue Id and get from, to, ip adresses Every thing goes right execpt that it happens “Queue ID collision” : for exemple in my logs : smtp_vm1:~$ cat /var/log/mail.log|grep 79E8223CA90D May 11 09:32:55 smtp_vm1 postfix/smtpd[2429]: 79E8223CA90D: client=unknown[XX.XX.XX.XX], sasl_method=LOGIN, sasl_username=XXX May 11 16:06:09 smtp_vm1 postfix/smtpd[13613]: 79E8223CA90D: client=XX.XX.XX.XX.rev.sfr.net[XX.XX.XX.XX], sasl_method=LOGIN, sasl_username=XX Same day, Same log file, 2 differents connexions, 2 différents login, same Queue ID... Is there any way to prevent this ? Thanks in advance Nico
Re: Queue Id Collision
Am 07.06.2012 14:26, schrieb Nico Angenon: I use Postfix 2.5.5 Every thing goes right execpt that it happens “Queue ID collision” Is there any way to prevent this? not with you terrible outdated version recent is 2.9.3 enable_long_queue_ids = yes http://www.postfix.org/postconf.5.html signature.asc Description: OpenPGP digital signature
Re: Queue Id Collision
* Nico Angenon n...@creaweb.fr: Hello, I use Postfix 2.5.5 on a linux debian box and i'm trying to make some statistics for sasl authenticated user... the goal is to show them how many connexions to the server they have been made, how many mails they have sent, and show them the detected from and to addresses... My search are based on Queue ID in the logs... finding a Queue Id, parse the log file to grep this Queue Id and get from, to, ip adresses Every thing goes right execpt that it happens “Queue ID collision” : for exemple in my logs : smtp_vm1:~$ cat /var/log/mail.log|grep 79E8223CA90D May 11 09:32:55 smtp_vm1 postfix/smtpd[2429]: 79E8223CA90D: client=unknown[XX.XX.XX.XX], sasl_method=LOGIN, sasl_username=XXX May 11 16:06:09 smtp_vm1 postfix/smtpd[13613]: 79E8223CA90D: client=XX.XX.XX.XX.rev.sfr.net[XX.XX.XX.XX], sasl_method=LOGIN, sasl_username=XX Same day, Same log file, 2 differents connexions, 2 différents login, same Queue ID... Is there any way to prevent this ? Update to a version (= 2.9) which has: enable_long_queue_ids -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Queue Id Collision
Nico Angenon: Every thing goes right execpt that it happens ?Queue ID collision? : for exemple in my logs : The Queue ID is not reused before a message is removed from the queue. Therefore there are no collisions. If you need unique queue IDs, use Postfix 2.8 long queue IDs Wietse
Re: Queue Id Collision
Hello, I've upgraded to postfix 2.9.1, put enable_long_queue_ids = yes in my /etc/postfix/main.cf file, and restarted postfix... Queue Id seems to be longger and uses all letters in the log file... Wait see... Thanks for all. Nico -Message d'origine- From: Wietse Venema Sent: Thursday, June 07, 2012 2:37 PM To: Nico Angenon Cc: postfix-users@postfix.org Subject: Re: Queue Id Collision Nico Angenon: Every thing goes right execpt that it happens ?Queue ID collision? : for exemple in my logs : The Queue ID is not reused before a message is removed from the queue. Therefore there are no collisions. If you need unique queue IDs, use Postfix 2.8 long queue IDs Wietse
Queue ID with amavisd
Hello Postfix Users :) I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPSno Queue ID Where is the Postfix queue ID? my master.cf: smtpd pass - - n - - smtpd -o smtpd_proxy_filter=127.0.0.1:10024 -o smtpd_client_connection_count_limit=10 -o smtpd_proxy_options=speed_adjust 127.0.0.1:10025 inet n - n - - smtpd -o smtpd_authorized_xforward_hosts=127.0.0.0/8 -o smtpd_client_restrictions= -o smtpd_helo_restrictions= -o smtpd_sender_restrictions= -o smtpd_recipient_restrictions=permit_mynetworks,reject -o smtpd_data_restrictions= -o mynetworks=127.0.0.0/8 -o receive_override_options=no_unknown_recipient_checks -- Chris
Re: Queue ID with amavisd
* Chris xchris...@googlemail.com: Hello Postfix Users :) I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPSno Queue ID Where is the Postfix queue ID? It's logged by the second smtpd, since the first smtpd using smtpd_proxy_filter doesn't issue an queueid -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Queue ID with amavisd
2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: Hello Postfix Users :) I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPS no Queue ID Where is the Postfix queue ID? It's logged by the second smtpd, since the first smtpd using smtpd_proxy_filter doesn't issue an queueid Can this be changed? -- Chris
Re: Queue ID with amavisd
* Chris xchris...@googlemail.com: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: Hello Postfix Users :) I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPS no Queue ID Where is the Postfix queue ID? It's logged by the second smtpd, since the first smtpd using smtpd_proxy_filter doesn't issue an queueid Can this be changed? Not without getting rid of smtpd_proxy_filter -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Queue ID with amavisd
2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: Hello Postfix Users :) I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPS no Queue ID Where is the Postfix queue ID? It's logged by the second smtpd, since the first smtpd using smtpd_proxy_filter doesn't issue an queueid Can this be changed? Not without getting rid of smtpd_proxy_filter Can I reject mails without smtpd_proxy_filter? -- Chris
Re: Queue ID with amavisd
* Chris xchris...@googlemail.com: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: Hello Postfix Users :) I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPS no Queue ID Where is the Postfix queue ID? It's logged by the second smtpd, since the first smtpd using smtpd_proxy_filter doesn't issue an queueid Can this be changed? Not without getting rid of smtpd_proxy_filter Can I reject mails without smtpd_proxy_filter? No. -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: Queue ID with amavisd
On Fri, Mar 02, 2012 at 05:32:18PM +0100, Chris wrote: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPS no Queue ID Where is the Postfix queue ID? It's logged by the second smtpd, since the first smtpd using smtpd_proxy_filter doesn't issue an queueid Can this be changed? Not without getting rid of smtpd_proxy_filter Can I reject mails without smtpd_proxy_filter? At this point you will do better if you back up and describe the problem you're trying to solve. Where/why do you need the queue ID displayed? -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if /dev/rob0 is in the Subject:
Re: Queue ID with amavisd
2012/3/2 /dev/rob0 r...@gmx.co.uk: On Fri, Mar 02, 2012 at 05:32:18PM +0100, Chris wrote: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPS no Queue ID Where is the Postfix queue ID? It's logged by the second smtpd, since the first smtpd using smtpd_proxy_filter doesn't issue an queueid Can this be changed? Not without getting rid of smtpd_proxy_filter Can I reject mails without smtpd_proxy_filter? At this point you will do better if you back up and describe the problem you're trying to solve. Where/why do you need the queue ID displayed? For diagnostic reasons. -- Chris
RE: Queue ID with amavisd
-Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix- us...@postfix.org] On Behalf Of Chris Sent: 02 March 2012 16:55 To: postfix-users@postfix.org Subject: Re: Queue ID with amavisd 2012/3/2 /dev/rob0 r...@gmx.co.uk: On Fri, Mar 02, 2012 at 05:32:18PM +0100, Chris wrote: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: 2012/3/2 Ralf Hildebrandt ralf.hildebra...@charite.de: * Chris xchris...@googlemail.com: I am using Postfix with amavisd. Received: from mail-wi0-f174.google.com (mail-wi0-f174.google.com [209.85.212.174]) by my.postfix-server.org (Postfix) with ESMTPS no Queue ID Where is the Postfix queue ID? It's logged by the second smtpd, since the first smtpd using smtpd_proxy_filter doesn't issue an queueid Can this be changed? Not without getting rid of smtpd_proxy_filter Can I reject mails without smtpd_proxy_filter? At this point you will do better if you back up and describe the problem you're trying to solve. Where/why do you need the queue ID displayed? For diagnostic reasons. -- Chris You could try implementing amavis using the content_filter parameter (after queue content filter). Instead of smtpd_proxy_filter (before queue content filter) Kind regards, James Day
postqueue -p | Queue ID | What does the '*' mean?
root@pinkie:/var/log# postqueue -p -Queue ID- --Size-- Arrival Time -Sender/Recipient--- 808AD1858E* 459339 Wed Nov 23 17:20:37 i...@dpsdirect.us @brookeagent.com 2BDBF18581* 459337 Wed Nov 23 17:50:25 i...@dpsdirect.us @brookeagent.com Does anyone know why these messages have a '*' after their ID? Thank You!
Re: postqueue -p | Queue ID | What does the '*' mean?
On Thu, 24 Nov 2011 02:03:17 +, peng...@sepserver.net wrote: root@pinkie:/var/log# postqueue -p -Queue ID- --Size-- Arrival Time -Sender/Recipient--- 808AD1858E* 459339 Wed Nov 23 17:20:37 i...@dpsdirect.us @brookeagent.com 2BDBF18581* 459337 Wed Nov 23 17:50:25 i...@dpsdirect.us @brookeagent.com Does anyone know why these messages have a '*' after their ID? Thank You! Ohhh I see those are in the active queue!
Re: long (non-repeating) queue ID support
On Sun, Mar 20, 2011 at 03:57:45PM -0400, Wietse Venema wrote: Below is the manpage entry for long queue ID support. Let me know if there's anything missing. With the first few characters of the new long queue-id encoding the epoch-seconds time and not the micro-seconds time, it seems to me that the current scheme for hashing the deferred and defer (and perhaps other) sub-queues needs adjustment. Also, just switching the position of the microseconds and epoch seconds in the new long queue-id is perhaps not quite enough, the first character of the new microseconds has only 8 possible values (0-7), previously the range was 16 values (more effective hashing). Use of base 32 encoding, would give a much better result, since 32^4 is 10^20 or just over a million, so the microseconds can be encoded just as efficiently in base 32 as in base 51, but the distribution of the first character is much more uniform (32 possible values). So my proposed update for the long queue id is: - 4 octets of base 32 encoded tv_usec - 6+ octets of base 51 encoded tv_sec epoch time - one non base 51 octet separator - inode number in base 52. -- Viktor.
Re: long (non-repeating) queue ID support
Wietse Venema: Victor Duchovni: On Sun, Mar 20, 2011 at 03:57:45PM -0400, Wietse Venema wrote: Below is the manpage entry for long queue ID support. Let me know if there's anything missing. With the first few characters of the new long queue-id encoding the epoch-seconds time and not the micro-seconds time, it seems to me that the current scheme for hashing the deferred and defer (and perhaps other) sub-queues needs adjustment. Also, just switching the position of the microseconds and epoch seconds in the new long queue-id is perhaps not quite enough, the first character of the new microseconds has only 8 possible values (0-7), previously the range was 16 values (more effective hashing). Use of base 32 encoding, would give a much better result, since 32^4 is 10^20 or just over a million, so the microseconds can be encoded just as efficiently in base 32 as in base 51, but the distribution of the first character is much more uniform (32 possible values). So my proposed update for the long queue id is: - 4 octets of base 32 encoded tv_usec - 6+ octets of base 51 encoded tv_sec epoch time - one non base 51 octet separator - inode number in base 52. Better: use reverse microseconds + reverse seconds (i.e. LSB first, base 52 encoded). Then we can have 52 possible values per character for queue hashing. With two levels of hashing we get 10x fewer files per directory compared to old queue file names. In fact, the existing base 52/51 ID is optimal when we use the last characters (the LSB end) for directory hashing. This is a near-trivial code change, and it spreads the queue evenly over 52 subdirectories. We'll also need some postcat command option to decode such information for forensic/trouble shooting purposes. Meaning, given a queue ID of 3PtcSh3sXLzHQDd, present it as date + microseconds + inode number if is a well-formed long Postfix queue ID (3PtcSh = Mar 21 08:50:12 2011). Wietse
Re: long (non-repeating) queue ID support
Victor Duchovni: On Sun, Mar 20, 2011 at 03:57:45PM -0400, Wietse Venema wrote: Below is the manpage entry for long queue ID support. Let me know if there's anything missing. With the first few characters of the new long queue-id encoding the epoch-seconds time and not the micro-seconds time, it seems to me that the current scheme for hashing the deferred and defer (and perhaps other) sub-queues needs adjustment. That is a very good point. Rather than overhauling the name scheme or using three different encodings for seconds, microseconds and inode number, I made a tiny change in the hash_queue_names algorithm. With long queue file names it now takes the (full time in seconds + three least significant characters from time in microseconds), then uses the last characters for hashing, so that: 3Ptdbd0p9qz1PFjk becomes q/9/3Ptdbd0p9qz1PFjk (here, 3Ptdbd = time in seconds, p9q = three LSB from microseconds). This way we can have meaningful hashing over more than three levels, with fewer files per directory because we have base 52 instead of base 16. I initially used the last characters from the whole file name (the inode number portion) but I was concerned that some systems may have unexpected structure in their inode numbers causing the hashing to become uneven (perhaps when some file system uses fake inode numbers that are really cookies of some kind). It also would not look pretty with dedicated queue file systems where inode numbers can be really small. Wietse
Re: long (non-repeating) queue ID support
On Mon, Mar 21, 2011 at 07:15:44AM -0400, Wietse Venema wrote: So my proposed update for the long queue id is: - 4 octets of base 32 encoded tv_usec - 6+ octets of base 51 encoded tv_sec epoch time - one non base 51 octet separator - inode number in base 52. Better: use reverse microseconds + reverse seconds (i.e. LSB first, base 52 encoded). Then we can have 52 possible values per character for queue hashing. With two levels of hashing we get 10x fewer files per directory compared to old queue file names. I am not sure that reversing the us value is better. With the us in big-endian format, the output directory is locally constant in time, so when a burst of mail arrives it is not overly scattered in multiple directories, while overall, the deferred queue is still split evenly. Also, with a little-endian base 52 queue-id, and hash depth of 2, we have 2704 directories to search, while big-endian 32 x 32 takes us to 977 directories as compared to 245 directories for big-endian base 16. We'll also need some postcat command option to decode such information for forensic/trouble shooting purposes. That may be useful. -- Viktor.
Re: long (non-repeating) queue ID support
On Mon, Mar 21, 2011 at 01:29:15PM -0400, Wietse Venema wrote: I thought of that. This is not a problem with today's default configuration where the high-traffic queues (i.e. incoming, active) are not hashed. Do you expect that there will be configurations that do hash their high-traffic queues? Scattering of queue files in the active queue is largely cost-less, it is rarely walked. The queue manager knows where all the files are. Essentially all active queue access is random access. The active queue should perhaps be hashed by default, this costs little and can help with large active queues. For the incoming queue, the queue-manager walks the queue pulling in the first file found (modulo a few incomplete files) and after re-start some files time-stamped into the future. So the queue manager is better off with an unhashed incoming queue, which also improves fairness, with traditional sequential directories. Of course cleanup adding files to a long incoming queue may be slowed down, this is perhaps not entirely a bad thing. When directories are btrees, files sort by name, and the locally constant hash directory + lexically monotone increasing queue ids lead to better fairness if incoming is hashed, but perhaps hashing of incoming should be discouraged. If so, perhaps we don't need to optimize for locally constant sub-directories. Also, with a little-endian base 52 queue-id, and hash depth of 2, we have 2704 directories to search, while big-endian 32 x 32 takes us to 977 directories as compared to 245 directories for big-endian base 16. Base 52 requires fewer levels of hashing than smaller bases, and 52 with depth 3 or more seems to make little sense. So that is a limitation in usability. I could just forget about lexicographical hashing and simply hash the hexadecimal representation of the microseconds (extracted from the queue file name and converted from base 52). With this there would be no change in file distribution compared to Postfix 2.8. This is an interesting idea, also, no new sub-directories in existing queues. Is it worth the effort though? The code gets complex when there is a mixture of 2.8 and 2.9 style files in the queue. -- Viktor.
Re: long (non-repeating) queue ID support
Victor Duchovni: Also, with a little-endian base 52 queue-id, and hash depth of 2, we have 2704 directories to search, while big-endian 32 x 32 takes us to 977 directories as compared to 245 directories for big-endian base 16. Base 52 requires fewer levels of hashing than smaller bases, and 52 with depth 3 or more seems to make little sense. So that is a limitation in usability. I could just forget about lexicographical hashing and simply hash the hexadecimal representation of the microseconds (extracted from the queue file name and converted from base 52). With this there would be no change in file distribution compared to Postfix 2.8. This is an interesting idea, also, no new sub-directories in existing queues. Is it worth the effort though? The code gets complex when there is a mixture of 2.8 and 2.9 style files in the queue. It's really trivial code, given that most of the work was already encapsulated in macros. We're dealing with trivially short strings, so CPU performance is not a concern either. I prefer the compactness of base52, but I don't like its behavior with lexical queue hashing. I'm checking the code as discussed above. Looks like it will be more text explaining things in the docs than actual code. Wietse
Re: long (non-repeating) queue ID support
Wietse Venema: I could just forget about lexicographical hashing and simply hash the hexadecimal representation of the microseconds (extracted from the queue file name and converted from base 52). With this there would be no change in file distribution compared to Postfix 2.8. This is an interesting idea, also, no new sub-directories in existing queues. Is it worth the effort though? The code gets complex when there is a mixture of 2.8 and 2.9 style files in the queue. It's really trivial code, given that most of the work was already encapsulated in macros. We're dealing with trivially short strings, so CPU performance is not a concern either. Implemented as postfix-20110321. The code change is only a few lines, less than the documentation change. Wietse 20110321 Performance: with long queue file names, queue hashing now produces the same result as with short names. Postfix uses the hexadecimal representation of the file creation time in microseconds, instead of the beginning of the file name which changes once every year or so, a problem that was reported by Victor Duchovni. The base 16 encoding gives finer control over the number of directories than possible with base 52 encoding. Files: global/mail_queue.[hc]. This change requires postfix reload.
long (non-repeating) queue ID support
Below is the manpage entry for long queue ID support. Let me know if there's anything missing. This code is part of this weekend's snapshot (*). Several iterations have been running on my systems through the past week. Wietse (*) As of Postfix 2.9, snapshot releases happen on weekends. enable_long_queue_ids (default: no) Enable long, non-repeating, queue IDs (queue file names). The benefit of non-repeating names is simpler logfile analysis and easier queue migration (there is no need to run postsuper to change queue file names that don't match their message file inode number). Note: see below for how to prepare long queue file names for migration to Postfix = 2.8. Changing the parameter value to yes has the following effects: o Existing queue file names are not affected. o New queue files are created with names such as 3Pt2mN2VXxznjll. These are encoded in a 52-character alphabet that contains dig- its (0-9), upper-case letters (B-Z) and lower-case letters (b- z). For safety reasons the vowels (AEIOUaeiou) are excluded from the alphabet. The name format is: 6 or more characters for the time in seconds, 4 characters for the time in microseconds, the 'z'; the remainder is the file inode number encoded in the first 51 characters of the 52-character alphabet. o New messages have a Message-ID header with queueID@myhostname. o The mailq (postqueue -p) output has a wider Queue ID column. The number of whitespace-separated fields is not changed. Changing the parameter value to no has the following effects: o Existing long queue file names are renamed to the short form (while running postfix reload or postsuper). o New queue files are created with names such as C3CD21F3E90 from a hexadecimal alphabet that contains digits (0-9) and upper-case letters (A-F). The name format is: 5 characters for the time in microseconds; the remainder is the file inode number. o New messages have a Message-ID header with MMDDHH- MMSS.queueid@myhostname, where MMDDHHMMSS are the year, month, day, hour, minute and second. o The mailq (postqueue -p) output has the same format as with Postfix = 2.8. Before migration to Postfix = 2.8, the following commands are required to convert long queue file names into short names: # postfix stop # postconf enable_long_queue_ids=no # postsuper Repeat the postsuper command until it reports no more queue file name changes.
Long queue ID support gotcha
Today I was testing an option for long Postfix queue IDs (queue file names) that won't be reused for about 30+ years. The idea is to prepend the 30 least significant bits of the time in seconds to the queue ID. For the long queue IDs I chose an encoding that packs more bits in a queue ID character. As a result, long queue IDs are typically only four characters longer than old queue IDs. Where old queue IDs use characters from 0-9A-F (4 bits per character, i.e. base 16 encoding), my first implementation of long queue IDs used characters from 0-9A-V (5 bits per character, i.e. base 32 encoding). The reason for using this base 32 character set was that it is compatible with Postfix's queue file name syntax checks, unlike base 64 which uses non-alphanumerics. This is helpful if you want to back out a change and go back to an older Postfix version, and not have all hell break loose because it rejects all the new queue files. Here is an example of the prototype logging: Mar 12 17:10:13 tail postfix/pickup[22915]: 6NNRQ6SKNI58V5S: uid=D from= Mar 12 17:10:13 tail postfix/cleanup[22921]: 6NNRQ6SKNI58V5S: message-id=20110312221013.6nnrq6skni58...@tail.porcupine.org Mar 12 17:10:13 tail postfix/qmgr[22916]: 6NNRQ6SKNI58V5S: from=a...@porcupine.org, size=328, nrcpt=1 (queue active) Mar 12 17:10:14 tail postfix/smtp[22923]: 6NNRQ6SKNI58V5S: to=aaa...@porcupine.org, orig_to=AA, relay=spike.porcupine.org[2001:240:587:0:2d0:b7ff:fe88:2ca7]:25, delay=0.07, delays=0.02/0.02/0.01/0.03, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as EEBB01F3E90) Mar 12 17:10:14 tail postfix/qmgr[22916]: 6NNRQ6SKNI58V5S: removed So far so good. Unfortunately, writing a secure mail system involves more than avoiding coding errors. The problem is that the larger alphabet and longer queue IDs increase the possibility that existing words will appear inside queue IDs (consider that the letters C, F, K and U are in the base 32 alphabet and that a queue ID can be 12 characters long). Even non-profane words may cause distraction. Real words can also appear with base64 encoding of message body parts, but most people aren't directly exposed to such information, so it is less of a problem. One solution is to eliminate the vowels (AEIOU) from the encoding, and to add one other character (or keep the A which was already used in old queue IDs) so I can keep packing 5 bits into each queue ID character. Another option is to use some encoding that is not a nice power of 2. Whatever the final implementation will be, I expect that support for long queue IDs will be disabled by default, at least initially. Wietse
Re: Long queue ID support gotcha
The idea is to prepend the 30 least significant bits of the time in seconds to the queue ID. Btw, 6 more hours to the next 'pretty' decimal unix timestamp: 13 Mark
Re: Long queue ID support gotcha
On 3/12/2011 7:31 PM, Wietse Venema wrote: [snip] The problem is that the larger alphabet and longer queue IDs increase the possibility that existing words will appear inside queue IDs (consider that the letters C, F, K and U are in the base 32 alphabet and that a queue ID can be 12 characters long). Even non-profane words may cause distraction. Real words can also appear with base64 encoding of message body parts, but most people aren't directly exposed to such information, so it is less of a problem. One solution is to eliminate the vowels (AEIOU) from the encoding, and to add one other character (or keep the A which was already used in old queue IDs) so I can keep packing 5 bits into each queue ID character. Another option is to use some encoding that is not a nice power of 2. [snip] But this precludes the possibility of having the $100 FACEDEADBEEF Mail ID generation contest. -Daniel
Exporting Postfix logging queue id to external processes started by pipe
Hello everybody, I use pipe to start my own script when an email to a certain address is received. The script will do it's own logging and I want to be able to correlate that logging with the Postifx logs. The queue id that is used in the Postifix logs seems ideal for that. I think it's called queue id in Postfix terminology, but to be clear, I'm referring to 21D5D497FE in the example below: Nov 11 12:29:39 WCEA006 postfix/smtpd[28104]: 21D5D497FE: client= So my question is: is there any way I can have pipe pass the queue id into my script? I couldn't find anything in the man page. If not, is there another way in which I can make an easy correlation to the Postifx logs? I use Postfix 2.7.0. Thank you, Catalin Iacob
Re: Exporting Postfix logging queue id to external processes started by pipe
On 11/15/2010 12:22 PM, Jeroen Geilman wrote: On 11/15/2010 11:55 AM, Catalin Iacob wrote: Hello everybody, I use pipe to start my own script when an email to a certain address is received. The script will do it's own logging and I want to be able to correlate that logging with the Postifx logs. The queue id that is used in the Postifix logs seems ideal for that. I think it's called queue id in Postfix terminology, but to be clear, I'm referring to 21D5D497FE in the example below: Nov 11 12:29:39 WCEA006 postfix/smtpd[28104]: 21D5D497FE: client= That is the Queue-ID, true. However, since it's used internally by postfix, once a message is delivered it is no longer of use. It is not communicated to external commands.So my question is: is there any way I can have pipe pass the queue id into my script? I couldn't find anything in the man page. If not, is there another way in which I can make an easy correlation to the Postifx logs? I use Postfix 2.7.0. Postfix logs delivery to a pipe(8) command by sepcifically naming the service in master.cf that was used. This provides one part - you can screen the postfix log for that transport. I would just log a (timestamp, sender, recipient, nexthop) tuple; this should provide enough specifics to correlate with the postfix logs. You can also log that same tuple for each message (on a single line) by adding smtpd_client_restrictions = check_client_access static:WARN Drat. That should probably have been smtp_recipient_restrictions = check_client_access static:WARN [, your other restrictions,...] or it won't log all information - although that may depend on the setting of smtpd_delay-reject. to your main.cf. Of course, the extra log comes at a marginal price in performance - I don't know how much exactly but it can't be big. Thank you, Catalin Iacob -- J.
Re: Exporting Postfix logging queue id to external processes started by pipe
Catalin Iacob: Hello everybody, I use pipe to start my own script when an email to a certain address is received. The script will do it's own logging and I want to be able to correlate that logging with the Postifx logs. The queue id that is used in the Postifix logs seems ideal for that. I think it's called queue id in Postfix terminology, but to be clear, I'm referring to 21D5D497FE in the example below: Nov 11 12:29:39 WCEA006 postfix/smtpd[28104]: 21D5D497FE: client= So my question is: is there any way I can have pipe pass the queue id into my script? I couldn't find anything in the man page. If not, is there another way in which I can make an easy correlation to the Postifx logs? I use Postfix 2.7.0. You MUST NOT manipulate the Postfix queue with other programs. Wietse
Re: Exporting Postfix logging queue id to external processes started by pipe
15.11.2010 14:59, Wietse Venema wrote: Catalin Iacob: Hello everybody, I use pipe to start my own script when an email to a certain address is received. The script will do it's own logging and I want to be able to correlate that logging with the Postifx logs. The queue id that is used in the Postifix logs seems ideal for that. I think it's called queue id in Postfix terminology, but to be clear, I'm referring to 21D5D497FE in the example below: Nov 11 12:29:39 WCEA006 postfix/smtpd[28104]: 21D5D497FE: client= So my question is: is there any way I can have pipe pass the queue id into my script? I couldn't find anything in the man page. If not, is there another way in which I can make an easy correlation to the Postifx logs? I use Postfix 2.7.0. You MUST NOT manipulate the Postfix queue with other programs. The question is not about manipulating the queue. The question is merely about logging and connecting various pieces in the log together. I, while used postfix alot in the past, now rarely perform new installs (it all just works :). Recently I moved our mail to a new server and redid all configuration again. And faced a very well-known (to me anyway) problem again: what's the easy way to see a message life-cycle in a mail system? While it is in postfix, it is all connected in the log using the same queueID, so grepping it will give you good picture. But once there's at least one external component, there's no such grouping anymore. Postfix itself, when doing certain operations, performs logging like this (from memory): date host postfix/local[123]: queueID: forwarded to f...@bar.com newID there you can see the newID on the same line as old queueID. When such forwarding is done by an external agent, we'll see two lines like this: date host forwarder[124]: msgid=baz-ms...@bar.com forwarded to f...@bar.com date host postfix/local[123]: queueID: b...@bar.com delivered to program (/some/where/forwarder) Not they only backwards (understandable), but they also aren't conected in a way which is easy to find. Sure, there should be no messing with postfix queue. But there should be a way to export queueID attribute when running external programs so that all delivery steps are easily greppable in the log. And this question is about the latter, not the former. /mjt
Queue-Id in content filter
Hi, I'm trying to write a pipe content filter for postfix. I have a question, Is There a way to put the queue id of email, to this content filter? It would be really useful to write a log file and trace the email flow. Thanks
Re: Queue-Id in content filter
On 12/15/2009 5:21 AM, Giovanni Mancuso wrote: Hi, I'm trying to write a pipe content filter for postfix. I have a question, Is There a way to put the queue id of email, to this content filter? No. It would be really useful to write a log file and trace the email flow. If you write your filter with SMTP support, you can use the XFORWARD extensions to help correlate your logs. http://www.postfix.org/XFORWARD_README.html -- Noel Jones
Re: Queue-Id in content filter
Giovanni Mancuso: Hi, I'm trying to write a pipe content filter for postfix. I have a question, Is There a way to put the queue id of email, to this content filter? It would be really useful to write a log file and trace the email flow. The queue ID is in the first RECEIVED: message header. Wietse
Queue id
Hi Checking the logs i've noticed that i have the same Queue Id for two connections... this could be? I Ask because i was looking for some specific entry and i found more than one entry Here is the log Feb 2 11:33:13 newcosmos postfix/smtpd[11863]: 54EF440BF: client=localhost[127.0.0.1] Feb 2 11:33:13 newcosmos postfix/cleanup[11953]: 54EF440BF: message-id=80f7634f1d414a629af611da1d975...@xxx.xx Feb 2 11:33:13 newcosmos postfix/qmgr[9654]: 54EF440BF: from=pvilc...@xxx.xx, size=51856, nrcpt=1 (queue active) Feb 2 11:33:15 newcosmos postfix/smtp[11938]: 54EF440BF: to=ptor...@xxx.xx, relay=127.0.0.1[127.0.0.1]:10024, delay=1.8, delays=0.11/0/0/1.7, dsn=2.6.0, status=sent (250 2.6.0 Ok, id=12437-02, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: as 103B640BD) Feb 2 11:33:15 newcosmos postfix/qmgr[9654]: 54EF440BF: removed Feb 2 13:44:24 newcosmos postfix/smtpd[26473]: 54EF440BF: client=unknown[172.21.20.70], sasl_method=LOGIN, sasl_username=personasv...@xxx.xx Feb 2 13:44:24 newcosmos postfix/cleanup[27320]: 54EF440BF: message-id=006101c98556$caadcfc0$60096f...@cl Feb 2 13:44:24 newcosmos postfix/qmgr[9654]: 54EF440BF: from=personasv...@xxx.xx, size=5299, nrcpt=1 (queue active) Feb 2 13:44:24 newcosmos postfix/lmtp[25587]: 54EF440BF: to=eduardo.ahum...@xxx.xx, relay=newcosmos.xxx.xx[/var/run/dspam/dspam.sock], delay=0.29, delays=0.06/0/0/0.23, dsn=2.6.0, status=sent (250 2.6.0 eduardo.ahum...@xxx.xx Message accepted for delivery) Feb 2 13:44:24 newcosmos postfix/qmgr[9654]: 54EF440BF: removed Thanks!!! Michael.-
Re: Queue id
Michael, From http://www.postfix.org/faq.html: Postfix names a queue file after its inode number and after the microsecond part of the time of day. Thus, if a queue file has a name based on someone elses inode number there is a small chance that the file name will collide with another queue file. The documentation warns in several places about the re-usability of queue IDs, e.g. in postsuper(1). -martin -- Martin Schmitt - Schmitt Systemberatung - http://www.scsy.de DE 35415 Pohlheim, Gießener Str. 18 DE 65307 Bad Schwalbach, Am Bräunchesberg 9 Linux/UNIX - Internet - E-Mail Infrastructure - Antispam/Antivirus - What goes up, must come down. Ask any system administrator. - signature.asc Description: OpenPGP digital signature
Re: Queue id
On Tue, 2009-02-03 at 17:09 +0100, Martin Schmitt (Schmitt Systemberatung) wrote: Michael, From http://www.postfix.org/faq.html: Postfix names a queue file after its inode number and after the microsecond part of the time of day. Thus, if a queue file has a name based on someone elses inode number there is a small chance that the file name will collide with another queue file. The documentation warns in several places about the re-usability of queue IDs, e.g. in postsuper(1). Well... Martin... Thanks for the input. Michael.- -martin
Re: Queue id
Michael Fernández M wrote: Hi Checking the logs i've noticed that i have the same Queue Id for two connections... this could be? I Ask because i was looking for some specific entry and i found more than one entry Here is the log The QueueID is not, and is not intended to be, a unique transaction identifier. The postfix QueueID is guaranteed to be unique only while the queue file exists; ie. only one file currently in the postfix queue may have any particular ID. Once the file is removed, that ID may be reused at any time. and fix your clock. -- Noel Jones
Re: Queue ID gets reused? Not unique?
Hi Victor, Perfect, thanks a lot! This is the information I was looking for. Durk On Thu, Nov 13, 2008 at 10:36:10PM +0100, Durk Strooisma wrote: I was examining my Postfix logs and saw two sequential sessions using the same queue ID. I was a bit surprised as I had the assumption that queue IDs were generated randomly, which means they should be practically unique. They are not random, which makes unique within: - The 1 second interval when the queue id is created, provided your clock does not jump backwards - The lifetime of the message that has that queue id When a new second stards, and the old message is gone, the queue id is available for re-use. Okay, so this could be a wrong assumption... My question is, how are queue IDs exactly generated? I couldn't find this info in the Postfix documentation, but I might have overlooked it. They are generated to avoid *collisions* of queue files names for messages that exist at the same time, but not otherwise intended to be unique beyond the two conditions above.
Re: Queue ID gets reused? Not unique?
I was examining my Postfix logs and saw two sequential sessions using the same queue ID. I was a bit surprised as I had the assumption that queue IDs were generated randomly, which means they should be practically unique. Postfix behaves as documented. Please point out where the documentation made the promise to you that queue IDs are unique. Thanks. Well, the documentation is fine. Actually, I think it's one of best among software projects. The only information I couldn't find was about the creation of queue IDs. Therefore I found myself in the situation I couldn't refute my assumption. Durk
Re: Queue ID gets reused? Not unique?
Durk Strooisma: I was examining my Postfix logs and saw two sequential sessions using the same queue ID. I was a bit surprised as I had the assumption that queue IDs were generated randomly, which means they should be practically unique. Postfix behaves as documented. Please point out where the documentation made the promise to you that queue IDs are unique. Thanks. Well, the documentation is fine. Actually, I think it's one of best among software projects. The only information I couldn't find was about the creation of queue IDs. Therefore I found myself in the situation I couldn't refute my assumption. Sometimes I am in the mood to pull people's leg. More seriously, I take pride in documenting the behavior that is guaranteed. The algorithm that assigns queue IDs may change, therefore the documentation makes no promises about how it's done. Currently the ID is the name of a short-lived file. A future queue implementation may use persistent files. In that case the queue ID may need to be assigned in some other way. The only hard requirement is that no two messages have the same ID while they are in the Postfix queue. Wietse
Re: Queue ID gets reused? Not unique?
Dear Wietse, thank you for your detailed explanation. In the future, would you consider having unique identifiers generated by an algorithm which would take into account the CPU ID (or other unique identifier), process ID time, so as to make it a unique ID worldwide, or is this not something which you would find to be of interest? I am asking this, in view of future possible instances of the law re: legal status of an email its authoritative tracking. Just curious. Thanks, Olivier - Original Message - From: Wietse Venema [EMAIL PROTECTED] To: Postfix users postfix-users@postfix.org Sent: Friday, November 14, 2008 12:40 PM Subject: Re: Queue ID gets reused? Not unique? Durk Strooisma: I was examining my Postfix logs and saw two sequential sessions using the same queue ID. I was a bit surprised as I had the assumption that queue IDs were generated randomly, which means they should be practically unique. Postfix behaves as documented. Please point out where the documentation made the promise to you that queue IDs are unique. Thanks. Well, the documentation is fine. Actually, I think it's one of best among software projects. The only information I couldn't find was about the creation of queue IDs. Therefore I found myself in the situation I couldn't refute my assumption. Sometimes I am in the mood to pull people's leg. More seriously, I take pride in documenting the behavior that is guaranteed. The algorithm that assigns queue IDs may change, therefore the documentation makes no promises about how it's done. Currently the ID is the name of a short-lived file. A future queue implementation may use persistent files. In that case the queue ID may need to be assigned in some other way. The only hard requirement is that no two messages have the same ID while they are in the Postfix queue. Wietse
Re: Queue ID gets reused? Not unique?
Olivier MJ Crepin-Leblond: Dear Wietse, thank you for your detailed explanation. In the future, would you consider having unique identifiers generated by an algorithm which would take into account the CPU ID (or other unique identifier), process ID time, so as to make it a unique ID worldwide, or is this not something which you would find to be of interest? The RFC822 Message-ID field is supposed to uniquely identify the message, not the transaction. This is easy to achieve when each MTA instance uses a different name. The RFC3463 3463 ENVID field is supposed to uniquely identify the transaction, not the message, for the purpose of delivery status notifications. The RFC3885 MTRK command provides a way to request message tracking based on the RFC3463 3463 ENVID field. Generally, it seems unrealistic to expect that Postfix will achieve global dominance, therefore the next best approach IMHO is to deploy an Internet standard that has global support. Given enough demand (or a sufficient quality design, in advance of implementation) Postfix may eventually support this. Wietse I am asking this, in view of future possible instances of the law re: legal status of an email its authoritative tracking. Just curious. Thanks, Olivier - Original Message - From: Wietse Venema [EMAIL PROTECTED] To: Postfix users postfix-users@postfix.org Sent: Friday, November 14, 2008 12:40 PM Subject: Re: Queue ID gets reused? Not unique? Durk Strooisma: I was examining my Postfix logs and saw two sequential sessions using the same queue ID. I was a bit surprised as I had the assumption that queue IDs were generated randomly, which means they should be practically unique. Postfix behaves as documented. Please point out where the documentation made the promise to you that queue IDs are unique. Thanks. Well, the documentation is fine. Actually, I think it's one of best among software projects. The only information I couldn't find was about the creation of queue IDs. Therefore I found myself in the situation I couldn't refute my assumption. Sometimes I am in the mood to pull people's leg. More seriously, I take pride in documenting the behavior that is guaranteed. The algorithm that assigns queue IDs may change, therefore the documentation makes no promises about how it's done. Currently the ID is the name of a short-lived file. A future queue implementation may use persistent files. In that case the queue ID may need to be assigned in some other way. The only hard requirement is that no two messages have the same ID while they are in the Postfix queue. Wietse
Queue ID gets reused? Not unique?
Hi all, I was examining my Postfix logs and saw two sequential sessions using the same queue ID. I was a bit surprised as I had the assumption that queue IDs were generated randomly, which means they should be practically unique. Okay, so this could be a wrong assumption... My question is, how are queue IDs exactly generated? I couldn't find this info in the Postfix documentation, but I might have overlooked it. Well, now some details for anyone interested in what happened. I'm running two machines (mail servers) with Debian 5.0 (lenny) and Postfix 2.5.5. Let's call them box A and box B. Box A was the machine using the same queue ID for two sessions. The accompanying log entries (and explanations): Box A (session 1): Nov 13 17:44:26 box-a postfix/smtpd[27915]: connect from localhost[127.0.0.1] Nov 13 17:44:26 box-a postfix/smtpd[27915]: 1C96531C9D: client=localhost[127.0.0.1] Nov 13 17:44:26 box-a postfix/cleanup[27917]: 1C96531C9D: message-id=[EMAIL PROTECTED] Nov 13 17:44:26 box-a postfix/qmgr[1917]: 1C96531C9D: from=[EMAIL PROTECTED], size=409, nrcpt=1 (queue active) Nov 13 17:44:26 box-a postfix/smtpd[27915]: disconnect from localhost[127.0.0.1] On the machine itself, [EMAIL PROTECTED] sends a mail to [EMAIL PROTECTED] Queued as 1C96531C9D. Nov 13 17:44:26 box-a postfix/smtp[27920]: 1C96531C9D: to=[EMAIL PROTECTED], relay=box-b.example.org[192.168.0.3]:25, delay=0, delays=0/0/0/0, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 1D0A81039FA5) Nov 13 17:44:26 box-a postfix/qmgr[1917]: 1C96531C9D: removed The mail can't be delivered locally, so is relayed to box B. Queued as 1D0A81039FA5. Box B (session 1): Nov 13 17:44:26 box-b postfix/smtpd[16249]: connect from box-a.example.org[192.168.0.2] Nov 13 17:44:26 box-b postfix/smtpd[16249]: 1D0A81039FA5: client=box-a.example.org[192.168.0.2] Nov 13 17:44:26 box-b postfix/cleanup[16251]: 1D0A81039FA5: message-id=[EMAIL PROTECTED] Nov 13 17:44:26 box-b postfix/qmgr[1893]: 1D0A81039FA5: from=[EMAIL PROTECTED], size=616, nrcpt=1 (queue active) Nov 13 17:44:26 box-b postfix/cleanup[16251]: 1E50E1039FB4: message-id=[EMAIL PROTECTED] Nov 13 17:44:26 box-b postfix/smtpd[16249]: disconnect from box-a.example.org[192.168.0.2] Mail is received from box A. Indeed queued as 1D0A81039FA5. Nov 13 17:44:26 box-b postfix/local[16252]: 1D0A81039FA5: to=[EMAIL PROTECTED], relay=local, delay=0.01, delays=0/0/0/0.01, dsn=2.0.0, status=sent (forwarded as 1E50E1039FB4) Nov 13 17:44:26 box-b postfix/qmgr[1893]: 1D0A81039FA5: removed There's an alias for bill on box B, so the mail is forwarded. Queued as 1E50E1039FB4. Nov 13 17:44:26 box-b postfix/qmgr[1893]: 1E50E1039FB4: from=[EMAIL PROTECTED], size=753, nrcpt=1 (queue active) Nov 13 17:44:26 box-b postfix/smtp[16253]: 1E50E1039FB4: to=[EMAIL PROTECTED], orig_to=[EMAIL PROTECTED], relay=box-a.example.org[192.168.0.2]:25, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 1C96531C9D) Nov 13 17:44:26 box-b postfix/qmgr[1893]: 1E50E1039FB4: removed The alias' target is on box A, so box B relays the mail to box A. Queued as 1C96531C9D. Hey, didn't we see this ID before? Box A (session 2): Nov 13 17:44:26 box-a postfix/smtpd[27915]: connect from box-b.example.org[192.168.0.3] Nov 13 17:44:26 box-a postfix/smtpd[27915]: 1C96531C9D: client=box-b.example.org[192.168.0.3] Nov 13 17:44:26 box-a postfix/cleanup[27917]: 1C96531C9D: message-id=[EMAIL PROTECTED] Nov 13 17:44:26 box-a postfix/qmgr[1917]: 1C96531C9D: from=[EMAIL PROTECTED], size=959, nrcpt=1 (queue active) Nov 13 17:44:26 box-a postfix/smtpd[27915]: disconnect from box-b.example.org[192.168.0.3] Nov 13 17:44:26 box-a postfix/virtual[27922]: 1C96531C9D: to=[EMAIL PROTECTED], relay=virtual, delay=0.01, delays=0/0/0/0.01, dsn=2.0.0, status=sent (delivered to maildir) Nov 13 17:44:26 box-a postfix/qmgr[1917]: 1C96531C9D: removed The mail is finally delivered. Indeed queued as 1C96531C9D. Yeah, we saw this ID before... in the beginning, when [EMAIL PROTECTED] sent the mail to the local Postfix daemon on the same machine. Some observations: - The reused queue ID is in a session that is in some way related to the first used of the queue ID. - The process is really fast, everything happens in the same second. - While replaying this scenario the duplicate queue ID isn't always reproducible. Like 2 times out of 10. I'm wondering if this behaviour of Postfix is normal. Thanks in advance for any information regarding this subeject! Durk
Re: Queue ID gets reused? Not unique?
On Thu, Nov 13, 2008 at 10:36:10PM +0100, Durk Strooisma wrote: I was examining my Postfix logs and saw two sequential sessions using the same queue ID. I was a bit surprised as I had the assumption that queue IDs were generated randomly, which means they should be practically unique. They are not random, which makes unique within: - The 1 second interval when the queue id is created, provided your clock does not jump backwards - The lifetime of the message that has that queue id When a new second stards, and the old message is gone, the queue id is available for re-use. Okay, so this could be a wrong assumption... My question is, how are queue IDs exactly generated? I couldn't find this info in the Postfix documentation, but I might have overlooked it. They are generated to avoid *collisions* of queue files names for messages that exist at the same time, but not otherwise intended to be unique beyond the two conditions above. -- Viktor. Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. To unsubscribe from the postfix-users list, visit http://www.postfix.org/lists.html or click the link below: mailto:[EMAIL PROTECTED] If my response solves your problem, the best way to thank me is to not send an it worked, thanks follow-up. If you must respond, please put It worked, thanks in the Subject so I can delete these quickly.
Re: Queue ID gets reused? Not unique?
Durk Strooisma: Hi all, I was examining my Postfix logs and saw two sequential sessions using the same queue ID. I was a bit surprised as I had the assumption that queue IDs were generated randomly, which means they should be practically unique. Postfix behaves as documented. Please point out where the documentation made the promise to you that queue IDs are unique. Wietse
