Re: message id is a unique number?
On Wed, 9 Mar 2011 12:57:26 + Mauro mrsan...@gmail.com wrote: I my logs I have: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. That number BF683A28247 is a unique number? Yes and no. It is unique in a timespan. If you use logrotate(8) it is probably unique for you, depending on your configuration. Cheers, Luciano. -- /\ /Via A. Salaino, 7 - 20144 Milano (Italy) \ / ASCII RIBBON CAMPAIGN / PHONE : +39 2 485781 FAX: +39 2 48578250 X AGAINST HTML MAIL/ E-MAIL: posthams...@sublink.sublink.org / \ AND POSTINGS/ WWW: http://www.mannucci.ORG/
Re: message id is a unique number?
Am 09.03.2011 13:57, schrieb Mauro: I my logs I have: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. That number BF683A28247 is a unique number? yes for this messages with cat /var/log/maillog | grep BF683A28247you get all lines from this message (sasl-user, from, to, deferrals...) signature.asc Description: OpenPGP digital signature
Re: message id is a unique number?
On 3/9/2011 6:57 AM, Mauro wrote: I my logs I have: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. That number BF683A28247 is a unique number? The postfix queueid identifies a single message while it's in the queue. The queueid is created from the queue file inode number and microsecond CPU time. The queueid is unique while that message exists; only one message at a time may have a specific queueid. Once the message exits the queue, that queueid can be reused at any time. I've seen a queueid reused within 30 minutes. Don't count on it being unique for any period of time. -- Noel Jones
Re: message id is a unique number?
On 9 March 2011 14:04, Noel Jones njo...@megan.vbhcs.org wrote: On 3/9/2011 6:57 AM, Mauro wrote: I my logs I have: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. That number BF683A28247 is a unique number? The postfix queueid identifies a single message while it's in the queue. The queueid is created from the queue file inode number and microsecond CPU time. The queueid is unique while that message exists; only one message at a time may have a specific queueid. Once the message exits the queue, that queueid can be reused at any time. I've seen a queueid reused within 30 minutes. Don't count on it being unique for any period of time. I need to know in one year who sent at who. I have logs for the year and records are like: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. Feb 13 06:28:13 mail1-xen postfix/qmgr[8336]: BF683A28247: to=.. What element I can use to identify who sent at who in the logs files?
Re: message id is a unique number?
* Mauro mrsan...@gmail.com: On 9 March 2011 14:04, Noel Jones njo...@megan.vbhcs.org wrote: On 3/9/2011 6:57 AM, Mauro wrote: I my logs I have: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. That number BF683A28247 is a unique number? The postfix queueid identifies a single message while it's in the queue. The queueid is created from the queue file inode number and microsecond CPU time. The queueid is unique while that message exists; only one message at a time may have a specific queueid. Once the message exits the queue, that queueid can be reused at any time. I've seen a queueid reused within 30 minutes. Don't count on it being unique for any period of time. I need to know in one year who sent at who. I have logs for the year and records are like: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. Feb 13 06:28:13 mail1-xen postfix/qmgr[8336]: BF683A28247: to=.. What element I can use to identify who sent at who in the logs files? Create you own tag. Use the WARN function in Postfix access (5) to generate a log entry. p@rick -- All technical questions asked privately will be automatically answered on the list and archived for public access unless privacy is explicitely required and justified. saslfinger (debugging SMTP AUTH): http://postfix.state-of-mind.de/patrick.koetter/saslfinger/
Re: message id is a unique number?
On 9 March 2011 15:46, Patrick Ben Koetter p...@state-of-mind.de wrote: * Mauro mrsan...@gmail.com: On 9 March 2011 14:04, Noel Jones njo...@megan.vbhcs.org wrote: On 3/9/2011 6:57 AM, Mauro wrote: I my logs I have: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. That number BF683A28247 is a unique number? The postfix queueid identifies a single message while it's in the queue. The queueid is created from the queue file inode number and microsecond CPU time. The queueid is unique while that message exists; only one message at a time may have a specific queueid. Once the message exits the queue, that queueid can be reused at any time. I've seen a queueid reused within 30 minutes. Don't count on it being unique for any period of time. I need to know in one year who sent at who. I have logs for the year and records are like: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. Feb 13 06:28:13 mail1-xen postfix/qmgr[8336]: BF683A28247: to=.. What element I can use to identify who sent at who in the logs files? Create you own tag. Use the WARN function in Postfix access (5) to generate a log entry. I have already logs of one year, I should parse these logs to identify who sent at who.
Re: message id is a unique number?
[root@mail:~]$ cat maillog | grep -i sasl | grep reindl | tail -n 2 Mar 9 15:00:22 mail postfix/smtpd[7582]: 0BA7FE9: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net Mar 9 16:23:45 mail postfix/smtpd[8877]: 614CEE8: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net [root@mail:~]$ cat maillog | grep 614CEE8 Mar 9 16:23:45 mail postfix/smtpd[8877]: 614CEE8: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net Mar 9 16:23:45 mail postfix/cleanup[8918]: 614CEE8: message-id=4d779b81.3050...@thelounge.net Mar 9 16:23:45 mail postfix/qmgr[19091]: 614CEE8: from=h.rei...@thelounge.net, size=2543, nrcpt=1 (queue active) Mar 9 16:23:45 mail postfix/lmtp[8303]: 614CEE8: to=strip...@thelounge.net, relay=127.0.0.1[127.0.0.1]:24, delay=0.15, delays=0.03/0/0/0.12, dsn=2.0.0, status=sent (215 Recipient strip...@thelounge.net OK) Mar 9 16:24:15 mail postfix/qmgr[19091]: 614CEE8: removed Am 09.03.2011 16:52, schrieb Mauro: On 9 March 2011 15:46, Patrick Ben Koetter p...@state-of-mind.de wrote: * Mauro mrsan...@gmail.com: On 9 March 2011 14:04, Noel Jones njo...@megan.vbhcs.org wrote: On 3/9/2011 6:57 AM, Mauro wrote: I my logs I have: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. That number BF683A28247 is a unique number? The postfix queueid identifies a single message while it's in the queue. The queueid is created from the queue file inode number and microsecond CPU time. The queueid is unique while that message exists; only one message at a time may have a specific queueid. Once the message exits the queue, that queueid can be reused at any time. I've seen a queueid reused within 30 minutes. Don't count on it being unique for any period of time. I need to know in one year who sent at who. I have logs for the year and records are like: Feb 13 06:27:57 mail1-xen postfix/qmgr[8336]: BF683A28247: from=.. Feb 13 06:28:13 mail1-xen postfix/qmgr[8336]: BF683A28247: to=.. What element I can use to identify who sent at who in the logs files? Create you own tag. Use the WARN function in Postfix access (5) to generate a log entry. I have already logs of one year, I should parse these logs to identify who sent at who. -- Mit besten Grüßen, Reindl Harald the lounge interactive design GmbH A-1060 Vienna, Hofmühlgasse 17 CTO / software-development / cms-solutions p: +43 (1) 595 3999 33, m: +43 (676) 40 221 40 icq: 154546673, http://www.thelounge.net/ signature.asc Description: OpenPGP digital signature
Re: message id is a unique number?
On 9 March 2011 16:19, Reindl Harald h.rei...@thelounge.net wrote: [root@mail:~]$ cat maillog | grep -i sasl | grep reindl | tail -n 2 Mar 9 15:00:22 mail postfix/smtpd[7582]: 0BA7FE9: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net Mar 9 16:23:45 mail postfix/smtpd[8877]: 614CEE8: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net [root@mail:~]$ cat maillog | grep 614CEE8 Mar 9 16:23:45 mail postfix/smtpd[8877]: 614CEE8: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net Mar 9 16:23:45 mail postfix/cleanup[8918]: 614CEE8: message-id=4d779b81.3050...@thelounge.net Mar 9 16:23:45 mail postfix/qmgr[19091]: 614CEE8: from=h.rei...@thelounge.net, size=2543, nrcpt=1 (queue active) Mar 9 16:23:45 mail postfix/lmtp[8303]: 614CEE8: to=strip...@thelounge.net, relay=127.0.0.1[127.0.0.1]:24, delay=0.15, delays=0.03/0/0/0.12, dsn=2.0.0, status=sent (215 Recipient strip...@thelounge.net OK) Mar 9 16:24:15 mail postfix/qmgr[19091]: 614CEE8: removed But from what I undestand 614CEE8 is not unique and I have to parse logs for one year.
Re: message id is a unique number?
On 3/9/2011 10:26 AM, Mauro wrote: On 9 March 2011 16:19, Reindl Haraldh.rei...@thelounge.net wrote: [root@mail:~]$ cat maillog | grep -i sasl | grep reindl | tail -n 2 Mar 9 15:00:22 mail postfix/smtpd[7582]: 0BA7FE9: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net Mar 9 16:23:45 mail postfix/smtpd[8877]: 614CEE8: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net [root@mail:~]$ cat maillog | grep 614CEE8 Mar 9 16:23:45 mail postfix/smtpd[8877]: 614CEE8: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net Mar 9 16:23:45 mail postfix/cleanup[8918]: 614CEE8: message-id=4d779b81.3050...@thelounge.net Mar 9 16:23:45 mail postfix/qmgr[19091]: 614CEE8: from=h.rei...@thelounge.net, size=2543, nrcpt=1 (queue active) Mar 9 16:23:45 mail postfix/lmtp[8303]: 614CEE8: to=strip...@thelounge.net, relay=127.0.0.1[127.0.0.1]:24, delay=0.15, delays=0.03/0/0/0.12, dsn=2.0.0, status=sent (215 Recipientstrip...@thelounge.net OK) Mar 9 16:24:15 mail postfix/qmgr[19091]: 614CEE8: removed But from what I undestand 614CEE8 is not unique and I have to parse logs for one year. counters for a specific queueid should be reset after a ... QUEUEID: removed log entry. -- Noel Jones
Re: message id is a unique number?
Noel Jones: On 3/9/2011 10:26 AM, Mauro wrote: On 9 March 2011 16:19, Reindl Haraldh.rei...@thelounge.net wrote: [root@mail:~]$ cat maillog | grep -i sasl | grep reindl | tail -n 2 Mar 9 15:00:22 mail postfix/smtpd[7582]: 0BA7FE9: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net Mar 9 16:23:45 mail postfix/smtpd[8877]: 614CEE8: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net [root@mail:~]$ cat maillog | grep 614CEE8 Mar 9 16:23:45 mail postfix/smtpd[8877]: 614CEE8: client=rh.thelounge.net[10.0.0.99], sasl_method=PLAIN, sasl_username=h.rei...@thelounge.net Mar 9 16:23:45 mail postfix/cleanup[8918]: 614CEE8: message-id=4d779b81.3050...@thelounge.net Mar 9 16:23:45 mail postfix/qmgr[19091]: 614CEE8: from=h.rei...@thelounge.net, size=2543, nrcpt=1 (queue active) Mar 9 16:23:45 mail postfix/lmtp[8303]: 614CEE8: to=strip...@thelounge.net, relay=127.0.0.1[127.0.0.1]:24, delay=0.15, delays=0.03/0/0/0.12, dsn=2.0.0, status=sent (215 Recipientstrip...@thelounge.net OK) Mar 9 16:24:15 mail postfix/qmgr[19091]: 614CEE8: removed But from what I undestand 614CEE8 is not unique and I have to parse logs for one year. counters for a specific queueid should be reset after a ... QUEUEID: removed log entry. Correct. With current Postfix implementations, there are two marker records that you can use: - The postfix/qmgr removed record that says the file is deleted. This record was introduced with Postfix version 2.1. - The postfix/smtpd ... client=... that says the file is created. This record is written by all Postfix versions. There is no equivalent record for mail that is submitted with the Postfix sendmail command. Instead use postfix/cleanup .. message-id=... which is also logged for SMTP mail. Wietse
Re: message id is a unique number?
On Wed, Mar 09, 2011 at 01:17:38PM -0500, Wietse Venema wrote: Correct. With current Postfix implementations, there are two marker records that you can use: - The postfix/qmgr removed record that says the file is deleted. This record was introduced with Postfix version 2.1. - The postfix/smtpd ... client=... that says the file is created. This record is written by all Postfix versions. There is no equivalent record for mail that is submitted with the Postfix sendmail command. Instead use postfix/cleanup .. message-id=... which is also logged for SMTP mail. In addition to qmqpd(8) logging message creation just like smtpd(8), in fact pickup(8) also logs message creation: 2011-03-09T12:55:01-05:00 amnesiac postfix/pickup[25191]: 27D602FB86: uid=52009 from=user Things get more interesting with internally generated messages, either indirect forwarding by local(8) or sender/postmaster notifications from ((sufficiently recent Postfix) bounce(8): 2011-03-09T13:23:18-05:00 amnesiac postfix/bounce[11606]: D55BD5049C4: sender non-delivery notification: BACC6504D20 these are logged after the cleanup(8) service logs the creation of the message and instead correlate to the processing of the old and new messages. These are not indicators that all previous instances of the new queue-id are unrelated. So there is a theoretical possibility that an smtpd(8) client=... log entry that goes with an aborted message delivery will get incorrectly associated with a non-SMTP internally generated message that reuses the queue id shortly after the aborted transaction. In practice, this is a non-issue, and the presense of bounce(8) or local(8) log entries can be used to pre-empt the association of the most recent instance of the new queue-id with any exteral source. -- Viktor.
Re: message id is a unique number?
Victor Duchovni: On Wed, Mar 09, 2011 at 01:17:38PM -0500, Wietse Venema wrote: Correct. With current Postfix implementations, there are two marker records that you can use: - The postfix/qmgr removed record that says the file is deleted. This record was introduced with Postfix version 2.1. - The postfix/smtpd ... client=... that says the file is created. This record is written by all Postfix versions. There is no equivalent record for mail that is submitted with the Postfix sendmail command. Instead use postfix/cleanup .. message-id=... which is also logged for SMTP mail. In addition to qmqpd(8) logging message creation just like smtpd(8), in fact pickup(8) also logs message creation: 2011-03-09T12:55:01-05:00 amnesiac postfix/pickup[25191]: 27D602FB86: uid=52009 from=user Things get more interesting with internally generated messages, either indirect forwarding by local(8) or sender/postmaster notifications from ((sufficiently recent Postfix) bounce(8): 2011-03-09T13:23:18-05:00 amnesiac postfix/bounce[11606]: D55BD5049C4: sender non-delivery notification: BACC6504D20 these are logged after the cleanup(8) service logs the creation of the message and instead correlate to the processing of the old and new messages. These are not indicators that all previous instances of the new queue-id are unrelated. So there is a theoretical possibility that an smtpd(8) client=... log entry that goes with an aborted message delivery will get incorrectly associated with a non-SMTP internally generated message that reuses the queue id shortly after the aborted transaction. In practice, this is a non-issue, and the presense of bounce(8) or local(8) log entries can be used to pre-empt the association of the most recent instance of the new queue-id with any exteral source. Perhaps it is time to replace the time-in-microseconds portion of the queue ID by a sufficient number of random bits. Wietse
Re: message id is a unique number?
On Wed, Mar 09, 2011 at 01:56:50PM -0500, Wietse Venema wrote: Perhaps it is time to replace the time-in-microseconds portion of the queue ID by a sufficient number of random bits. I would not replace the microsecond time, its monotonicity has useful properties. Rather, we could augment the microsecond time and inode with ~16 additional bits, say cleanup appends to the microsecond encoding, before the inode: (epoch time 0xff) 8 | (pid + msg count) 0xff On a lightly loaded system with a single cleanup doing all the work, the pid + msg count will be locally monotone even if the clock drifts back. While pid + msg count collisions will happen on busy systems, the clock should keep repetitions at least 256 seconds apart, but in practice the odds of the microseconds and pid also colliding when the same inode is being re-used are extremely low. -- Viktor.
RE: message id is a unique number?
For what it's worth, sendmail's implementation encodes the current time down to the second plus the pid of the handling process in its queue IDs. A collision then could only happen if the same pid got re-used twice in the same second. It doesn't include the inode or any random data. Details: http://www.ale.org/pipermail/ale/2001-May/022331.html Similar to the issue of log correlation, in the OpenDKIM stats project work we had to have an SQL key across the reporting host, queue ID and timestamp columns to account for the fact that postfix recycles queue IDs, sometimes relatively quickly. -MSK
Re: message id is a unique number?
Murray S. Kucherawy: For what it's worth, sendmail's implementation encodes the current time down to the second plus the pid of the handling process in its queue IDs. A collision then could only happen if the same pid got re-used twice in the same second. It doesn't include the inode or any random data. Details: http://www.ale.org/pipermail/ale/2001-May/022331.html Similar to the issue of log correlation, in the OpenDKIM stats project work we had to have an SQL key across the reporting host, queue ID and timestamp columns to account for the fact that postfix recycles queue IDs, sometimes relatively quickly. There is one difference: Sendmail can just pick a name, and pick another one if the name already exists in a particular directory. Postfix uses the inode number in the name, because the name needs to be unique across the incoming, active, and deferred directories. Postfix could lengthen the time before reuse, by including more time information (four hex digits for ~1 day, six hex digits for ~0.5 year, eight hex digits for ~100 years). Seven hex digits should be sufficient to silence any complaints. Tighter packing is possible, but we're restricted to letters and digits (i.e. base 62 math). Wietse
Re: message id is a unique number?
On Wed, Mar 09, 2011 at 04:05:18PM -0500, Wietse Venema wrote: Postfix uses the inode number in the name, because the name needs to be unique across the incoming, active, and deferred directories. Postfix could lengthen the time before reuse, by including more time information (four hex digits for ~1 day, six hex digits for ~0.5 year, eight hex digits for ~100 years). Seven hex digits should be sufficient to silence any complaints. Tighter packing is possible, but we're restricted to letters and digits (i.e. base 62 math). Couldn't one also freely use _ and + for a complete base64 alphabet? Certainly log parsers would have to adapt, but is there another reason? -- Viktor.
Re: message id is a unique number?
Victor Duchovni wrote: On Wed, Mar 09, 2011 at 04:05:18PM -0500, Wietse Venema wrote: Postfix uses the inode number in the name, because the name needs to be unique across the incoming, active, and deferred directories. Postfix could lengthen the time before reuse, by including more time information (four hex digits for ~1 day, six hex digits for ~0.5 year, eight hex digits for ~100 years). Seven hex digits should be sufficient to silence any complaints. Tighter packing is possible, but we're restricted to letters and digits (i.e. base 62 math). Couldn't one also freely use _ and + for a complete base64 alphabet? Certainly log parsers would have to adapt, but is there another reason? time since EPOCH?
Re: message id is a unique number?
Victor Duchovni: On Wed, Mar 09, 2011 at 04:05:18PM -0500, Wietse Venema wrote: Postfix uses the inode number in the name, because the name needs to be unique across the incoming, active, and deferred directories. Postfix could lengthen the time before reuse, by including more time information (four hex digits for ~1 day, six hex digits for ~0.5 year, eight hex digits for ~100 years). Seven hex digits should be sufficient to silence any complaints. Tighter packing is possible, but we're restricted to letters and digits (i.e. base 62 math). Couldn't one also freely use _ and + for a complete base64 alphabet? Certainly log parsers would have to adapt, but is there another reason? Breaking logfile parsers might be one. The Postfix queue file module has strict checks on queue file name syntax. I had to add permission to use _ for flush(8) logs, but I would rather not water down the syntax restrictions further. Wietse
