Re: [Declude.JunkMail] Log Corruption
Title: Message
Sorry, I guess I misunderstood twice in this regard. I suppose that
for now, the old format is fine and the logging is better than before
being that it is only once per message instead of once per each
recipient. A normalized single line format would be preferable
overall, however I can think of many other things that would take
precedence on my wish list at the moment.
Matt
Andy Schmidt wrote:
but why change the logs to single
lines and not at the same time seek to normalize the format
They were not changed to single lines?
Single linesof ONE message were grouped
together - rather than scattered about. They are still single lines -
just the ORDER of their appearance in the log has changed. They are no
longer sorted strictly by "time" but rather by message.
May be I misunderstand what you are looking at?
Best Regards
Andy Schmidt
HM Systems Software, Inc.
600 East Crescent Avenue, Suite 203
Upper Saddle River, NJ 07458-1846
Phone: +1 201 934-3414
x20 (Business)
Fax: +1 201 934-9206
http://www.HM-Software.com/
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]] On Behalf Of Matt
Sent: Tuesday, March 01, 2005 04:54 PM
To: Declude.JunkMail@declude.com
Subject: Re: [Declude.JunkMail] Log Corruption
I guess this is somewhat rhetorical at this point, but why change the
logs to single lines and not at the same time seek to normalize the
format using the same standards that are used for Web logs or even a
simple comma separated (quoted qualifier) database format?
Matt
--
=
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=
RE: [Declude.JunkMail] Log Corruption
Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were clean, but considering it only happens a few times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Log Corruption
I had the opposite experience. Back at, oh, 1.7x I ran on LOGLEVEL HIGH, and had lots of log corruption. I had to drop down to MID. The increase in spam volume made it such that at MID, I had lots of log corruption again. With 2.x and the lines being written in a batch, I noticed an immediate drop in my disk usage and a huge drop in corrupted lines. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andy Schmidt Sent: Tuesday, March 01, 2005 9:22 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were clean, but considering it only happens a few times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Log Corruption
Yesterday's report on my declude logfile showed =...10.01% 02/28/2005..10.01% FF156FB0094D05A.10.01% I00510.01% QCEA370970074D08A...10.01% SET.10.01% SNIFFQCE053ABD0086D062..10.01% This is from Total Messages Processed: 12,316 And loglevel set to HIGH and the size of the file is 122 MB I am running 2.0.5 Goran Jovanovic The LAN Shoppe -Original Message- From: [EMAIL PROTECTED] [mailto:Declude.JunkMail- [EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, March 01, 2005 2:33 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption I had the opposite experience. Back at, oh, 1.7x I ran on LOGLEVEL HIGH, and had lots of log corruption. I had to drop down to MID. The increase in spam volume made it such that at MID, I had lots of log corruption again. With 2.x and the lines being written in a batch, I noticed an immediate drop in my disk usage and a huge drop in corrupted lines. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andy Schmidt Sent: Tuesday, March 01, 2005 9:22 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were clean, but considering it only happens a few times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Log Corruption
There has always been corruption of the logs. What seems to have changed here is that there is a new log format and the programmed exceptions in DLanalyzer are no longer able to handle the types of corruption as seamlessly as in the past. I haven't seen the new log format, but I would imagine that if they built it in a column format and kept it to a single line, similar to Web logging standards, it should be even easier to parse the data and correct for issues. If they didn't keep to consistent columns, then this could be just as big of an issue to parse as before and a lot of the exception rules would have to be rewritten for the new format. Matt Goran Jovanovic wrote: Yesterday's report on my declude logfile showed =...10.01% 02/28/2005..10.01% FF156FB0094D05A.10.01% I00510.01% QCEA370970074D08A...10.01% SET.10.01% SNIFFQCE053ABD0086D062..10.01% This is from Total Messages Processed: 12,316 And loglevel set to HIGH and the size of the file is 122 MB I am running 2.0.5 Goran Jovanovic The LAN Shoppe -Original Message- From: [EMAIL PROTECTED] [mailto:Declude.JunkMail- [EMAIL PROTECTED]] On Behalf Of Colbeck, Andrew Sent: Tuesday, March 01, 2005 2:33 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption I had the opposite experience. Back at, oh, 1.7x I ran on LOGLEVEL HIGH, and had lots of log corruption. I had to drop down to MID. The increase in spam volume made it such that at MID, I had lots of log corruption again. With 2.x and the lines being written in a batch, I noticed an immediate drop in my disk usage and a huge drop in corrupted lines. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Andy Schmidt Sent: Tuesday, March 01, 2005 9:22 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were "clean", but considering it only happens a "few" times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [Declude.JunkMail] Log Corruption
It's the same log format except that all the messages for a specific email are grouped together. Well most of the time they are grouped together. It does make eyeball parsing of the log easier. I load the logs into a database daily. With the 1.8x versions, I'll need to code around a new type of log corruption every one to two months. For the 3 days I was running 2.x, I had to code around 4 new corruption issues. Visually there is also more corruption. Odd things is that I never get log corruption in message sniffer. So somebody is writing out logs that don't get corrupt. - Original Message - From: Matt To: Declude.JunkMail@declude.com Sent: Tuesday, March 01, 2005 3:16 PM Subject: Re: [Declude.JunkMail] Log Corruption There has always been corruption of the logs. What seems to have changed here is that there is a new log format and the programmed exceptions in DLanalyzer are no longer able to handle the types of corruption as seamlessly as in the past.I haven't seen the new log format, but I would imagine that if they built it in a column format and kept it to a single line, similar to Web logging standards, it should be even easier to parse the data and correct for issues. If they didn't keep to consistent columns, then this could be just as big of an issue to parse as before and a lot of the exception rules would have to be rewritten for the new format.MattGoran Jovanovic wrote: Yesterday's report on my declude logfile showed =...10.01% 02/28/2005..10.01% FF156FB0094D05A.10.01% I00510.01% QCEA370970074D08A...10.01% SET.10.01% SNIFFQCE053ABD0086D062..10.01% This is from Total Messages Processed: 12,316 And loglevel set to HIGH and the size of the file is 122 MB I am running 2.0.5 Goran Jovanovic The LAN Shoppe -Original Message- From: [EMAIL PROTECTED] [mailto:Declude.JunkMail- [EMAIL PROTECTED]] On Behalf Of Colbeck, Andrew Sent: Tuesday, March 01, 2005 2:33 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption I had the opposite experience. Back at, oh, 1.7x I ran on LOGLEVEL HIGH, and had lots of log corruption. I had to drop down to MID. The increase in spam volume made it such that at MID, I had lots of log corruption again. With 2.x and the lines being written in a batch, I noticed an immediate drop in my disk usage and a huge drop in corrupted lines. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Andy Schmidt Sent: Tuesday, March 01, 2005 9:22 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were "clean", but considering it only happens a "few" times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. -- = Mail
RE: [Declude.JunkMail] Log Corruption
Title: Message I haven't seen the new log format I'm not aware of any new log format. The FORMAT of the log files stayed the same (regular "version" specificchanges may apply) on the ORDER changes. Meaning: all log lines for aSINGLE messageare now appearing in one set of lines - instead of log lines for various different messages appearing in strict "time" order. Before it was a "time" log, now it's more of a "message" log - but the log lines stay the same. Best RegardsAndy SchmidtHM Systems Software, Inc.600 East Crescent Avenue, Suite 203Upper Saddle River, NJ 07458-1846Phone: +1 201 934-3414 x20 (Business)Fax: +1 201 934-9206http://www.HM-Software.com/ -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of MattSent: Tuesday, March 01, 2005 04:16 PMTo: Declude.JunkMail@declude.comSubject: Re: [Declude.JunkMail] Log CorruptionThere has always been corruption of the logs. What seems to have changed here is that there is a new log format and the programmed exceptions in DLanalyzer are no longer able to handle the types of corruption as seamlessly as in the past.I haven't seen the new log format, but I would imagine that if they built it in a column format and kept it to a single line, similar to Web logging standards, it should be even easier to parse the data and correct for issues. If they didn't keep to consistent columns, then this could be just as big of an issue to parse as before and a lot of the exception rules would have to be rewritten for the new format.MattGoran Jovanovic wrote: Yesterday's report on my declude logfile showed =...10.01% 02/28/2005..10.01% FF156FB0094D05A.10.01% I00510.01% QCEA370970074D08A...10.01% SET.10.01% SNIFFQCE053ABD0086D062..10.01% This is from Total Messages Processed: 12,316 And loglevel set to HIGH and the size of the file is 122 MB I am running 2.0.5 Goran Jovanovic The LAN Shoppe -Original Message- From: [EMAIL PROTECTED] [mailto:Declude.JunkMail- [EMAIL PROTECTED]] On Behalf Of Colbeck, Andrew Sent: Tuesday, March 01, 2005 2:33 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption I had the opposite experience. Back at, oh, 1.7x I ran on LOGLEVEL HIGH, and had lots of log corruption. I had to drop down to MID. The increase in spam volume made it such that at MID, I had lots of log corruption again. With 2.x and the lines being written in a batch, I noticed an immediate drop in my disk usage and a huge drop in corrupted lines. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Andy Schmidt Sent: Tuesday, March 01, 2005 9:22 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were "clean", but considering it only happens a "few" times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and ty
RE: [Declude.JunkMail] Log Corruption
Title: Message Odd things is that I never get log corruption in message sniffer. So somebody is writing out logs that don't get corrupt. Maybe because he is running as a "service", thus can serialize the log output?
Re: [Declude.JunkMail] Log Corruption
Title: Message I guess this is somewhat rhetorical at this point, but why change the logs to single lines and not at the same time seek to normalize the format using the same standards that are used for Web logs or even a simple comma separated (quoted qualifier) database format? Matt Andy Schmidt wrote: I haven't seen the new log format I'm not aware of any new log format. The FORMAT of the log files stayed the same (regular "version" specificchanges may apply) on the ORDER changes. Meaning: all log lines for aSINGLE messageare now appearing in one set of lines - instead of log lines for various different messages appearing in strict "time" order. Before it was a "time" log, now it's more of a "message" log - but the log lines stay the same. Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax: +1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Tuesday, March 01, 2005 04:16 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Log Corruption There has always been corruption of the logs. What seems to have changed here is that there is a new log format and the programmed exceptions in DLanalyzer are no longer able to handle the types of corruption as seamlessly as in the past. I haven't seen the new log format, but I would imagine that if they built it in a column format and kept it to a single line, similar to Web logging standards, it should be even easier to parse the data and correct for issues. If they didn't keep to consistent columns, then this could be just as big of an issue to parse as before and a lot of the exception rules would have to be rewritten for the new format. Matt Goran Jovanovic wrote: Yesterday's report on my declude logfile showed =...10.01% 02/28/2005..10.01% FF156FB0094D05A.10.01% I00510.01% QCEA370970074D08A...10.01% SET.10.01% SNIFFQCE053ABD0086D062..10.01% This is from Total Messages Processed: 12,316 And loglevel set to HIGH and the size of the file is 122 MB I am running 2.0.5 Goran Jovanovic The LAN Shoppe -Original Message- From: [EMAIL PROTECTED] [mailto:Declude.JunkMail- [EMAIL PROTECTED]] On Behalf Of Colbeck, Andrew Sent: Tuesday, March 01, 2005 2:33 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption I had the opposite experience. Back at, oh, 1.7x I ran on LOGLEVEL HIGH, and had lots of log corruption. I had to drop down to MID. The increase in spam volume made it such that at MID, I had lots of log corruption again. With 2.x and the lines being written in a batch, I noticed an immediate drop in my disk usage and a huge drop in corrupted lines. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Andy Schmidt Sent: Tuesday, March 01, 2005 9:22 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were "clean", but considering it only happens a "few" times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type "unsubscribe Declude.JunkMail". The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (ht
Re: [Declude.JunkMail] Log Corruption
Title: Message Because it wasn't needed for Smartermail. - Original Message - From: Matt To: Declude.JunkMail@declude.com Sent: Tuesday, March 01, 2005 3:54 PM Subject: Re: [Declude.JunkMail] Log Corruption I guess this is somewhat rhetorical at this point, but why change the logs to single lines and not at the same time seek to normalize the format using the same standards that are used for Web logs or even a simple comma separated (quoted qualifier) database format?MattAndy Schmidt wrote: I haven't seen the new log format I'm not aware of any new log format. The FORMAT of the log files stayed the same (regular "version" specificchanges may apply) on the ORDER changes. Meaning: all log lines for aSINGLE messageare now appearing in one set of lines - instead of log lines for various different messages appearing in strict "time" order. Before it was a "time" log, now it's more of a "message" log - but the log lines stay the same. Best RegardsAndy SchmidtHM Systems Software, Inc.600 East Crescent Avenue, Suite 203Upper Saddle River, NJ 07458-1846Phone: +1 201 934-3414 x20 (Business)Fax: +1 201 934-9206http://www.HM-Software.com/ -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of MattSent: Tuesday, March 01, 2005 04:16 PMTo: Declude.JunkMail@declude.comSubject: Re: [Declude.JunkMail] Log CorruptionThere has always been corruption of the logs. What seems to have changed here is that there is a new log format and the programmed exceptions in DLanalyzer are no longer able to handle the types of corruption as seamlessly as in the past.I haven't seen the new log format, but I would imagine that if they built it in a column format and kept it to a single line, similar to Web logging standards, it should be even easier to parse the data and correct for issues. If they didn't keep to consistent columns, then this could be just as big of an issue to parse as before and a lot of the exception rules would have to be rewritten for the new format.MattGoran Jovanovic wrote: Yesterday's report on my declude logfile showed =...10.01% 02/28/2005..10.01% FF156FB0094D05A.10.01% I00510.01% QCEA370970074D08A...10.01% SET.10.01% SNIFFQCE053ABD0086D062..10.01% This is from Total Messages Processed: 12,316 And loglevel set to HIGH and the size of the file is 122 MB I am running 2.0.5 Goran Jovanovic The LAN Shoppe -Original Message- From: [EMAIL PROTECTED] [mailto:Declude.JunkMail- [EMAIL PROTECTED]] On Behalf Of Colbeck, Andrew Sent: Tuesday, March 01, 2005 2:33 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption I had the opposite experience. Back at, oh, 1.7x I ran on LOGLEVEL HIGH, and had lots of log corruption. I had to drop down to MID. The increase in spam volume made it such that at MID, I had lots of log corruption again. With 2.x and the lines being written in a batch, I noticed an immediate drop in my disk usage and a huge drop in corrupted lines. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Andy Schmidt Sent: Tuesday, March 01, 2005 9:22 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were "clean", but considering it only happens a "few" times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, jus
Re: [Declude.JunkMail] Log Corruption
There has always been corruption of the logs. What seems to have changed here is that there is a new log format and the programmed exceptions in DLanalyzer are no longer able to handle the types of corruption as seamlessly as in the past. The log format has not changed other than blocking specific messages together. The issue about the log corruption has existed in Declude for as long as I can remember. In regards to DLAnalyzer it makes every best effort to determine if the log file line that was corrupted is salvageable. However, if you look at the log files and you will see from time to time the corruption will look like this. 04/14/2004 23:59:59 Qb45c04ab008c9990 Msg failed SPAM04/14/2004 23:59:59 Where the start of a new line will show up the middle of a test name. Due to this type of corruption it is very difficult when the user is running in auto-detect test name modes to know if SPAM04/14/2004 is a valid test name or not. Techinically it could be.. The one thing to keep in mind if you would like to phase these types of corrupted tests out of your reports you can switch to the manual test mode where you define which tests are valid on your system. There is also an import option that you can use where DLAnalyzer will read your $default$.junkmail and import in your tests. Darrell Goran Jovanovic wrote: Yesterday's report on my declude logfile showed =...10.01% 02/28/2005..10.01% FF156FB0094D05A.10.01% I00510.01% QCEA370970074D08A...10.01% SET.10.01% SNIFFQCE053ABD0086D062..10.01% This is from Total Messages Processed: 12,316 And loglevel set to HIGH and the size of the file is 122 MB I am running 2.0.5 Goran Jovanovic The LAN Shoppe -Original Message- From: [EMAIL PROTECTED] [mailto:Declude.JunkMail- [EMAIL PROTECTED] On Behalf Of Colbeck, Andrew Sent: Tuesday, March 01, 2005 2:33 PM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption I had the opposite experience. Back at, oh, 1.7x I ran on LOGLEVEL HIGH, and had lots of log corruption. I had to drop down to MID. The increase in spam volume made it such that at MID, I had lots of log corruption again. With 2.x and the lines being written in a batch, I noticed an immediate drop in my disk usage and a huge drop in corrupted lines. Andrew 8) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andy Schmidt Sent: Tuesday, March 01, 2005 9:22 AM To: Declude.JunkMail@declude.com Subject: RE: [Declude.JunkMail] Log Corruption Hi, Yesterday was actually a good day (here a snippet): TEST # FAILED Percentage 18:51:5810.01% SNIF02/28/2005..10.01% SPA02/28/2005...10.01% WEIG02/28/2005..10.01% Sure, I used to get reports that were clean, but considering it only happens a few times a day, I don't consider it a major issue and I can live with it. Also I used to run in MID log level, but after 1.76 (or so), they removed the test summary line form MID level, and I needed to switch to HIGH log level. It's entirely possible that the log corruption is not really specific to 2.0x - but rather a secondary result because they forded me to deal with tremendously large log files (vs. the old MID log files). Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software
Re: [Declude.JunkMail] Log Corruption
I though Pete had some locking mechanism built in to prevent overlapping. Pete? Darrell Andy Schmidt writes: Odd things is that I never get log corruption in message sniffer. So somebody is writing out logs that don't get corrupt. Maybe because he is running as a service, thus can serialize the log output? Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Log Corruption
Title: Message but why change the logs to single lines and not at the same time seek to normalize the format They were not changed to single lines? Single linesof ONE message were grouped together - rather than scattered about. They are still single lines - just the ORDER of their appearance in the log has changed. They are no longer sorted strictly by "time" but rather by message. May be I misunderstand what you are looking at? Best RegardsAndy SchmidtHM Systems Software, Inc.600 East Crescent Avenue, Suite 203Upper Saddle River, NJ 07458-1846Phone: +1 201 934-3414 x20 (Business)Fax: +1 201 934-9206http://www.HM-Software.com/ -Original Message-From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of MattSent: Tuesday, March 01, 2005 04:54 PMTo: Declude.JunkMail@declude.comSubject: Re: [Declude.JunkMail] Log CorruptionI guess this is somewhat rhetorical at this point, but why change the logs to single lines and not at the same time seek to normalize the format using the same standards that are used for Web logs or even a simple comma separated (quoted qualifier) database format?Matt attachment: HMSoftSmall.jpg
Re: [Declude.JunkMail] Log Corruption
I think Pete's still in the bunker with the shields on high. - Original Message - From: Darrell ([EMAIL PROTECTED]) [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, March 01, 2005 4:38 PM Subject: Re: [Declude.JunkMail] Log Corruption I though Pete had some locking mechanism built in to prevent overlapping. Pete? Darrell Andy Schmidt writes: Odd things is that I never get log corruption in message sniffer. So somebody is writing out logs that don't get corrupt. Maybe because he is running as a service, thus can serialize the log output? Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
RE: [Declude.JunkMail] Log Corruption
(Pete isn't here much) I remember this thread from a long time back. Messsage Sniffer doesn't take any particular efforts to lock the log file to prevent collisions. And he agreed that Microsoft Windows had the nasty habit of not always serializing writeln statements to a text file. Scott wasn't inclined to add a spinlock feature so that the log lines could be cleaned up, on the bases that it was effort taken away from other priorities, that corruption tended to indicate a struggling server that ought to be upgraded, and general annoyance that calling an atomic function in the OS... didn't work. Andrew 8( -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Darrell ([EMAIL PROTECTED]) Sent: Tuesday, March 01, 2005 2:39 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Log Corruption I though Pete had some locking mechanism built in to prevent overlapping. Pete? Darrell Andy Schmidt writes: Odd things is that I never get log corruption in message sniffer. So somebody is writing out logs that don't get corrupt. Maybe because he is running as a service, thus can serialize the log output? Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] Log Corruption
I disagree with the struggling server logic. We saw the log corruption in a test environment a year ago that had minimal traffic, say a couple thousand messages a day. It was a dual 1.4GHz processor with 1 GB RAM and 10k RPM SCSI drives. Load was only about 1-5% during testing. Darin. - Original Message - From: Colbeck, Andrew [EMAIL PROTECTED] To: Declude.JunkMail@declude.com Sent: Tuesday, March 01, 2005 5:48 PM Subject: RE: [Declude.JunkMail] Log Corruption (Pete isn't here much) I remember this thread from a long time back. Messsage Sniffer doesn't take any particular efforts to lock the log file to prevent collisions. And he agreed that Microsoft Windows had the nasty habit of not always serializing writeln statements to a text file. Scott wasn't inclined to add a spinlock feature so that the log lines could be cleaned up, on the bases that it was effort taken away from other priorities, that corruption tended to indicate a struggling server that ought to be upgraded, and general annoyance that calling an atomic function in the OS... didn't work. Andrew 8( -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Darrell ([EMAIL PROTECTED]) Sent: Tuesday, March 01, 2005 2:39 PM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Log Corruption I though Pete had some locking mechanism built in to prevent overlapping. Pete? Darrell Andy Schmidt writes: Odd things is that I never get log corruption in message sniffer. So somebody is writing out logs that don't get corrupt. Maybe because he is running as a service, thus can serialize the log output? Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
