Re: Integrating the logs
ISOQLOG looks terrific, but: 1) It solves a different problem than the one I am trying to solve. It appears to provide traffic summaries. 2) The problem I am trying to solve is to create a detailed trace from start to finish about the entire life cycle of a single email message. I dislike s**dmail just as much as everyone else here, but being a monolithic program, it did have a detailed trace for a given message. I'm fine with qmail being broken up into components, but I need an integrated log for debugging. 3) There are several posts in the archives about isoqlog not running if the multilog is under 100k and not rotating. I am only testing now, so my multilogs are quite small. The solutions are not clear to me. One is to find a patch for multilog which rotates the logfile, whenever it receives a HUP. A) where is the patch? B) exactly how am I supposed to supply the HUP in my scripts? From: hari_bhr [EMAIL PROTECTED] Reply-To: hari_bhr [EMAIL PROTECTED] To: pop corn [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: Integrating the logs Date: Thu, 28 Jun 2001 10:54:03 +0530 look for ISOQLOG nice - Original Message - From: pop corn [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, June 28, 2001 10:00 AM Subject: Integrating the logs I have my different daemons logging into the various log subdirectories via multilog. My problem now is integrating them so that I have a continuous line of activity from the beginning to end for a given email. For example, I can do a tail -f current log for qmail-pop3 while running tests. However, I would like to know what related activities are occurring in other logs for this same email test. I have pulled the following info about qmail-analog from the following length thread in the archives. It includes an example script. I cut/paste quickly, so not everyone gets the credit they deserve for their posts in this thread. I have at minimum two questions after reading all of the info below: 1) what are all the z... files in the example script? for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay; 2) where is a real working example of qmail-mrtg? == I want to know how many messages were sent/failed etc. for a given period of time (say the last three days). I have done the following in both /var/log/qmail/qmail-send and /var/log/qmail/qmail-smtpd (I'll admit my ignorance and say that I don't know the difference between the two. Is qmail-send local deliveries and qmail-smtpd remote deliveries?): 1) Ran matchup on /var/log/qmail/qmail-send(smtpd)/current 2) Converted the matchedup version of current into human readable format using tai64nlocal 3) Pulled out dates for which I want to see log results from the file created above 4) Convert the data above to tai64 format using tai64n 5) Ran this data through zoverall to see qmailanalog results Regardless of whether I run it against /var/log/qmail/qmail-send or /var/log/qmail/qmail-smtpd I get the following: Completed messages: 0 Total delivery attempts: 0 Am I anywhere near doing this right? Here are my actual commands 1) cat /var/log/qmail/qmail-smtpd/current | /usr/local/qmailanalog/bin/matchup /var/log/qmail/qmail-smtpd/matchedup 2) cat /var/log/qmail/qmail-smtpd/matchedup | /usr/local/bin/tai64nlocal human_readable_current 3) vi human_readable_current (remove all unneeded data) 4) cat /var/log/qmail/qmail-send/human_readable_current | /usr/local/bin/tai64n tai64_current 5) cat ./tai64_current | /usr/local/qmailanalog/bin/zoverall overall_log No. qmail-smtpd is incoming mail via SMTP. qmail-send is all deliveries, local and remote. No. Instead of converting the tai64n timestamps to human-readable, you need to convert them to the fractional seconds (tai) that qmail-analog expects. You can do this with tai64n2tai, included in Bruce Guenter's qlogtools package if I remember correctly. His software is at untroubled.org. Thanks for the info Charles, but I'm confused. How do most of you folks pull out information from your logs? Log files generated by qmail are unreadable/unusable in the current (multilog) format. In order for them to make sense to me, and in order to sift them for specific dates I have to convert them to human readable format. I can do this with tai64nlocal. Once I have removed data that is not pertinent I then have to change them back into multilog format using tai64n, and then convert them into the older TAI64 format that qmailanalog understands, then run them through the qmailanalog scripts. Wow, that's a convoluted process using tools that until now had worked together to provide a graceful solution to my email needs. Thanks for the info Charles, but I'm confused. How do most of you folks pull out information from your logs? With qmail
Re: Integrating the logs
pop corn [EMAIL PROTECTED] wrote: 1) what are all the z... files in the example script? for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay; These are the actual statistical analysis programs from qmail-analog. See the documentation in qmail-analog for details about what they do, and the reports they generate. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ ---
Re: Integrating the logs
These are the actual statistical analysis programs from qmail-analog. See the documentation in qmail-analog for details about what they do, and the reports they generate. Will do! _ Get your FREE download of MSN Explorer at http://explorer.msn.com
Integrating the logs
I have my different daemons logging into the various log subdirectories via multilog. My problem now is integrating them so that I have a continuous line of activity from the beginning to end for a given email. For example, I can do a tail -f current log for qmail-pop3 while running tests. However, I would like to know what related activities are occurring in other logs for this same email test. I have pulled the following info about qmail-analog from the following length thread in the archives. It includes an example script. I cut/paste quickly, so not everyone gets the credit they deserve for their posts in this thread. I have at minimum two questions after reading all of the info below: 1) what are all the z... files in the example script? for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay; 2) where is a real working example of qmail-mrtg? == I want to know how many messages were sent/failed etc. for a given period of time (say the last three days). I have done the following in both /var/log/qmail/qmail-send and /var/log/qmail/qmail-smtpd (I'll admit my ignorance and say that I don't know the difference between the two. Is qmail-send local deliveries and qmail-smtpd remote deliveries?): 1) Ran matchup on /var/log/qmail/qmail-send(smtpd)/current 2) Converted the matchedup version of current into human readable format using tai64nlocal 3) Pulled out dates for which I want to see log results from the file created above 4) Convert the data above to tai64 format using tai64n 5) Ran this data through zoverall to see qmailanalog results Regardless of whether I run it against /var/log/qmail/qmail-send or /var/log/qmail/qmail-smtpd I get the following: Completed messages: 0 Total delivery attempts: 0 Am I anywhere near doing this right? Here are my actual commands 1) cat /var/log/qmail/qmail-smtpd/current | /usr/local/qmailanalog/bin/matchup /var/log/qmail/qmail-smtpd/matchedup 2) cat /var/log/qmail/qmail-smtpd/matchedup | /usr/local/bin/tai64nlocal human_readable_current 3) vi human_readable_current (remove all unneeded data) 4) cat /var/log/qmail/qmail-send/human_readable_current | /usr/local/bin/tai64n tai64_current 5) cat ./tai64_current | /usr/local/qmailanalog/bin/zoverall overall_log No. qmail-smtpd is incoming mail via SMTP. qmail-send is all deliveries, local and remote. No. Instead of converting the tai64n timestamps to human-readable, you need to convert them to the fractional seconds (tai) that qmail-analog expects. You can do this with tai64n2tai, included in Bruce Guenter's qlogtools package if I remember correctly. His software is at untroubled.org. Thanks for the info Charles, but I'm confused. How do most of you folks pull out information from your logs? Log files generated by qmail are unreadable/unusable in the current (multilog) format. In order for them to make sense to me, and in order to sift them for specific dates I have to convert them to human readable format. I can do this with tai64nlocal. Once I have removed data that is not pertinent I then have to change them back into multilog format using tai64n, and then convert them into the older TAI64 format that qmailanalog understands, then run them through the qmailanalog scripts. Wow, that's a convoluted process using tools that until now had worked together to provide a graceful solution to my email needs. Thanks for the info Charles, but I'm confused. How do most of you folks pull out information from your logs? With qmail-analog, tai64nlocal, and less, in my case. Most people here probably use something similar. Log files generated by qmail are unreadable/unusable in the current (multilog) format. tai64n timestamps aren't supposed to be human readable. They're supposed to be easily parsable by programs. That's the whole point of tai64nlocal -- you log with tai64n timestamps, and if you want to read the log with human-readable timestamps, you do: tai64nlocal log | pager_of_choice Don't run the logs through tai64nlocal before they hit the disk. In order for them to make sense to me, and in order to sift them for specific dates I have to convert them to human readable format. No, it's much simpler than that. A program to filter a log with tai64nlocal timestamps for particular dates is trivial; Bruce's qlogtools probably includes one (though I haven't checked). After you've filtered them, you run it through tai64nlocal before reading it. Once I have removed data that is not pertinent I then have to change them back into multilog format using tai64n, and then convert them into the older TAI64 format that qmailanalog understands, then run them through the qmailanalog scripts. Don't remove any data. What isn't pertinent? qmail-analog needs all of the various data that qmail-send logs to be able to accurately summarize it. I have a script that runs every night to give me a summary of the day's
Re: Integrating the logs
look for ISOQLOG nice - Original Message - From: pop corn [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, June 28, 2001 10:00 AM Subject: Integrating the logs I have my different daemons logging into the various log subdirectories via multilog. My problem now is integrating them so that I have a continuous line of activity from the beginning to end for a given email. For example, I can do a tail -f current log for qmail-pop3 while running tests. However, I would like to know what related activities are occurring in other logs for this same email test. I have pulled the following info about qmail-analog from the following length thread in the archives. It includes an example script. I cut/paste quickly, so not everyone gets the credit they deserve for their posts in this thread. I have at minimum two questions after reading all of the info below: 1) what are all the z... files in the example script? for ana in zoverall zddist zdeferrals zfailures zrhosts zsuids zrxdelay; 2) where is a real working example of qmail-mrtg? == I want to know how many messages were sent/failed etc. for a given period of time (say the last three days). I have done the following in both /var/log/qmail/qmail-send and /var/log/qmail/qmail-smtpd (I'll admit my ignorance and say that I don't know the difference between the two. Is qmail-send local deliveries and qmail-smtpd remote deliveries?): 1) Ran matchup on /var/log/qmail/qmail-send(smtpd)/current 2) Converted the matchedup version of current into human readable format using tai64nlocal 3) Pulled out dates for which I want to see log results from the file created above 4) Convert the data above to tai64 format using tai64n 5) Ran this data through zoverall to see qmailanalog results Regardless of whether I run it against /var/log/qmail/qmail-send or /var/log/qmail/qmail-smtpd I get the following: Completed messages: 0 Total delivery attempts: 0 Am I anywhere near doing this right? Here are my actual commands 1) cat /var/log/qmail/qmail-smtpd/current | /usr/local/qmailanalog/bin/matchup /var/log/qmail/qmail-smtpd/matchedup 2) cat /var/log/qmail/qmail-smtpd/matchedup | /usr/local/bin/tai64nlocal human_readable_current 3) vi human_readable_current (remove all unneeded data) 4) cat /var/log/qmail/qmail-send/human_readable_current | /usr/local/bin/tai64n tai64_current 5) cat ./tai64_current | /usr/local/qmailanalog/bin/zoverall overall_log No. qmail-smtpd is incoming mail via SMTP. qmail-send is all deliveries, local and remote. No. Instead of converting the tai64n timestamps to human-readable, you need to convert them to the fractional seconds (tai) that qmail-analog expects. You can do this with tai64n2tai, included in Bruce Guenter's qlogtools package if I remember correctly. His software is at untroubled.org. Thanks for the info Charles, but I'm confused. How do most of you folks pull out information from your logs? Log files generated by qmail are unreadable/unusable in the current (multilog) format. In order for them to make sense to me, and in order to sift them for specific dates I have to convert them to human readable format. I can do this with tai64nlocal. Once I have removed data that is not pertinent I then have to change them back into multilog format using tai64n, and then convert them into the older TAI64 format that qmailanalog understands, then run them through the qmailanalog scripts. Wow, that's a convoluted process using tools that until now had worked together to provide a graceful solution to my email needs. Thanks for the info Charles, but I'm confused. How do most of you folks pull out information from your logs? With qmail-analog, tai64nlocal, and less, in my case. Most people here probably use something similar. Log files generated by qmail are unreadable/unusable in the current (multilog) format. tai64n timestamps aren't supposed to be human readable. They're supposed to be easily parsable by programs. That's the whole point of tai64nlocal -- you log with tai64n timestamps, and if you want to read the log with human-readable timestamps, you do: tai64nlocal log | pager_of_choice Don't run the logs through tai64nlocal before they hit the disk. In order for them to make sense to me, and in order to sift them for specific dates I have to convert them to human readable format. No, it's much simpler than that. A program to filter a log with tai64nlocal timestamps for particular dates is trivial; Bruce's qlogtools probably includes one (though I haven't checked). After you've filtered them, you run it through tai64nlocal before reading it. Once I have removed data that is not pertinent I then have to change them back into multilog format using tai64n, and then convert them into the older TAI64 format that qmailanalog understands, then run them through the qmailanalog
