RE: AW: AW: How to change queue id?
Hello Wietse, ok, here is my problem in detail: I pass mails to Postfix through smtpd. Postfix sends them out and returns the queue_id. I save the queue_id to a database (UPDATE table SET mail_id = '...' WHERE id = ...). in a table structured like this: ID (INT, auto increment, indexed) - unique id for internal purpose mail_id (VARCHAR) - the queue_id from Postfix html (TEXT) - the html part of the message text (TEXT) - the text part of the message recipient (VARCHAR) - the email address of the recipient status (VARCHAR) - the status of the email (can be sent or bounced, when the message bounced, it saves the dsn and reason as well) I parse the mail.log for bounces every hour. When I find a bounce in the log I look up the mail_id in the db table and set the status to bounced (UPDATE table SET status = bounced WHERE mail_id = '...'). So for every bounce I need two database queries: 1. The update to save the queue_id 2. The update to set the status to bounced This is not efficient and performs badly when the table becomes bigger because the second query takes a long time. Reasons: 1. Text compare in WHERE clause 2. no index on mail_id I cannot set an index on the mail_id since that would slow down the inserting process to much (believe me, I have tried that option). So that was the problem. And now here is what I thought would be an adequate solution: Since I already have a unique id in the table I could use that instead of the mail_id. It would eliminate query no. 1 and speed up query no. 2 by an order-of-magnitude since the WHERE clause would only include one indexed column (id). So I thought I could find a way with Postfix to pass my internal table id to the mail.log when a bounce is logged and then parse it after. That's why I tried changing the queue_id to my internal id and when that didn't work tried to set the MAIL FROM extension to the internal id but it is not logged in the same line making the parsing unreliable (the mail from and the bounce notification are connected through the queue_id but are logged on different lines that could end up in different files when parsed every hour). I hope the explanation of the problem helped to find a solution, Best, Steffen -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Wietse Venema Sent: Sonntag, 7. Oktober 2012 13:59 To: Postfix users Subject: Re: AW: AW: How to change queue id? Steffen Schebesta: Hello Witse, I really don't want to get on your nerves but as far as I understand using the standardized bounce messages from Postfix Start explaining the problem. Stop talking about what you think are solutions. Stop wasting everyone's time on this mailing list. You have an application that generates email. You have an MTA that tries to deliver that email. Now what is the problem? Removing obsolete addresses from the list? Stop talking about solutions like changing the queue ID or changing the Postfix logging. Wietse
AW: AW: How to change queue id?
Using Wietse's first approach (adding a custom id to the MAIL FROM address as an extension) I have tried to output the sender's address in the same line of the mail.log as the bounce message. I believe I would need to change the global/log_adhoc.c (e.g. line 109 in 2.9.4) to do so but I cannot access the sender there. Is there a way to access the initial sender from the log_adhoc void? Maybe through the RECIPIENT object: RECIPIENT-QMGR_QUEUE-QMGR_ENTRY_LIST-QMGR_ENTRY-QMGR_MESSAGE-sender I know it sounds awefully complicated but I really need a custom id in the same line as the bounce message. Alternatively, if there is another location where I access and ouput the sender, the recipient, the status=bounced (reason) and the dsn that would also work. Any help is greatly appreciated. Best, Steffen -Ursprüngliche Nachricht- Von: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] Im Auftrag von Wietse Venema Gesendet: Donnerstag, 4. Oktober 2012 14:26 An: Postfix users Betreff: Re: AW: How to change queue id? Steffen Schebesta: If that doesn't work though then maybe I could work around this problem. I thought about adding the message-id to the bounce message but that probably Postfix has lots of options to identify a returned message. 1) You can encode the unique identifier in the envelope sender address (as an address extension). This works even when mail is returned by a remote MTA. 2) Your unique Message ID is already returned in the bounce message. See RFC 3462. This may or may not work for mail that is returned by a remote MTA. 3) You can specify your own unique ENVELOPE ID via SMTP (see RFC 3461) or via the Postfix sendmail -V option. This ENVELOPE ID is also returned in the bounce message. See RFC 3464. This works only if all MTAs in the forward path implement RFC 3461. I suggest that you study the many RFCs and non-RFC features that Postfix already implements, before tinkering with internals. But first, as Stan mentioned, you should describe your problem, instead of your proposed solutions. There are likely better solutions that already exist. I already mentioned three of them. Wietse
RE: AW: How to change queue id?
Hello Wietse, option 1 is actually a pretty good idea - hadn't thought of this. The disadvantage of handling the return mail is that not only bounces are returned but also automatic responders, etc. and thus I would have to parse the content of the email to find out if it is a real bounce. So I think parsing the mail.log is actually much easier... Options 2 and 3 are interesting ideas as well but - as you said yourself - aren't reliable enough. So in the end I still think a custom field in the mail.log when the bounce is reported would make most sense. Steffen -Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of Wietse Venema Sent: Donnerstag, 4. Oktober 2012 14:26 To: Postfix users Subject: Re: AW: How to change queue id? Steffen Schebesta: If that doesn't work though then maybe I could work around this problem. I thought about adding the message-id to the bounce message but that probably Postfix has lots of options to identify a returned message. 1) You can encode the unique identifier in the envelope sender address (as an address extension). This works even when mail is returned by a remote MTA. 2) Your unique Message ID is already returned in the bounce message. See RFC 3462. This may or may not work for mail that is returned by a remote MTA. 3) You can specify your own unique ENVELOPE ID via SMTP (see RFC 3461) or via the Postfix sendmail -V option. This ENVELOPE ID is also returned in the bounce message. See RFC 3464. This works only if all MTAs in the forward path implement RFC 3461. I suggest that you study the many RFCs and non-RFC features that Postfix already implements, before tinkering with internals. But first, as Stan mentioned, you should describe your problem, instead of your proposed solutions. There are likely better solutions that already exist. I already mentioned three of them. Wietse
How to change queue id?
Hello everybody, I deliver mails to my Postfix through smtpd. Postfix then takes it and sends it out to the recipient. Now I'm trying to change the queue_id for each email in Postfix 2.9 source code so that it is equal to the Message-ID (it is unique, don't worry) that I set in the email header when passing the email to Postfix through smtpd. I've looked in the source code for hours but cannot find the first time queue_id is set. In smtpd.c that (in my understanding) handles incoming mail through smtpd the variable state-queue_id is already used and only read but not written. I would really appreciate any tips where I can find the initial setting and on how I can set state-queue_id to Message-ID Thanks, Steffen
AW: How to change queue id?
Thanks for all the insightful answers. So, I actually use the long_queue_ids options and I save the queue_ids to a database to later compare them to the queue_ids found in the mail log to parse and mark the bounces. The problem - and thus the source for my question - is that this always means a string compare in the database search which is aweful for performance. So my thought was, to always pass a unique Message-ID to postfix equal to the autoincrement ID in my database, and then set this Message-ID as the mail's queue_id. This way the lookup in the database would be much faster plus I wouldn't have to save the queue_id in the database in the first place (huge performance boost). In my case I can guarantee that the Message-ID is always passed and that it is unique, so that wouldn't be the problem. But if Postfix creates the queue file name before receiving the message content then it actually is a bigger problem than I thought, except if I could change the queue_id or append the id to the queue_id... If that doesn't work though then maybe I could work around this problem. I thought about adding the message-id to the bounce message but that probably won't work because the message-id would not be in Postfix anymore at the time the bounce happens. Any other options that I could try? Really appreciate your help here! Best, Steffen -Ursprüngliche Nachricht- Von: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] Im Auftrag von Wietse Venema Gesendet: Mittwoch, 3. Oktober 2012 18:47 An: Postfix users Betreff: Re: How to change queue id? Steffen Schebesta: Hello everybody, I deliver mails to my Postfix through smtpd. Postfix then takes it and sends it out to the recipient. Now I'm trying to change the queue_id for each email in Postfix 2.9 source code so that it is equal to the Message-ID (it is unique, don't worry) that I set in the email header when passing the email to Postfix through smtpd. I've looked in the source code for hours but cannot find the first time queue_id is set. In smtpd.c that (in my understanding) handles incoming mail through smtpd the variable state-queue_id is already used and only read but not written. I would really appreciate any tips where I can find the initial setting and on how I can set state-queue_id to Message-ID Reasons not to use the message-id as the queue file name: - Different messages may contain the same Message-ID value. - Some messages don't contain a Message-ID. This field is not required by current RFC documents. - Chicken-and-egg problem: Postfix chooses the queue file name (and creates the file) before it receives the message content. I suggest that you set enable_long_queue_ids = yes. Long queue ID strings don't repeat as long as your clock moves forward. Wietse
