Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs
Confirmed! The new version starts up nice and quick On Mon, Oct 4, 2021 at 4:56 AM Doug Lytle wrote: > On 10/3/21 7:53 PM, K Post wrote: > > Thanks Doug. How long does it take to start the rebuild SpamDB worker > > though? Mine also runs 11, and seemingly fine, it just takes a > > long time to start. I don't know why it starts before the GUI or > > SMTP, but suspect that it needs to be running if it's going to track > > mail for the rebuild as it arrives. > > Looks like Thomas may have fixed your startup problem, > > Doug > > > ___ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs
On 10/3/21 7:53 PM, K Post wrote: Thanks Doug. How long does it take to start the rebuild SpamDB worker though? Mine also runs 11, and seemingly fine, it just takes a long time to start. I don't know why it starts before the GUI or SMTP, but suspect that it needs to be running if it's going to track mail for the rebuild as it arrives. Looks like Thomas may have fixed your startup problem, Doug ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs
Thanks Doug. How long does it take to start the rebuild SpamDB worker though? Mine also runs 11, and seemingly fine, it just takes a long time to start. I don't know why it starts before the GUI or SMTP, but suspect that it needs to be running if it's going to track mail for the rebuild as it arrives. On Fri, Oct 1, 2021 at 3:13 PM Doug Lytle wrote: > >>> I've been messing around (I don't think I broke anything), but at > command line "starting rebuild SpamDB worker thread" sits for 3+ minutes. > > Kevin, > > Restarting my install shows > > Starting rebuild SpamDB worker thread [10001] - ThreadCycleTime is set to > 30 microseconds > > I'm running under Devuan 3 Linux > > Doug > > > ___ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs
>>> I've been messing around (I don't think I broke anything), but at command >>> line "starting rebuild SpamDB worker thread" sits for 3+ minutes. Kevin, Restarting my install shows Starting rebuild SpamDB worker thread [10001] - ThreadCycleTime is set to 30 microseconds I'm running under Devuan 3 Linux Doug ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs
I've been messing around (I don't think I broke anything), but at command line "starting rebuild SpamDB worker thread" sits for 3+ minutes. Is that normal? So I understand, why does the rebuild SpamDB worker thread start at startup? Does it do things that early where it doesn't make sense to first have SMTP and the GUI operational first? On Wed, Sep 29, 2021 at 2:33 PM K Post wrote: > Thank you Thomas!! I will run through these steps this weekend and report > back. > > On Wed, Sep 29, 2021 at 12:03 PM Thomas Eckardt < > thomas.ecka...@thockar.com> wrote: > >> - check all the setting for the lists and hashes to be set to 'DB:' >> (using mysql) >> - check that the mysql server is working correctly (if not assp will >> failover to plain files) >> - check all settings related to the rebuildspamdb >> - shutdown assp >> - clear the folder 'tmpDB' >> - remove all 'griplist' related files from assp root folder >> - check the file time of the assp.cfg (written at assp shutdown) >> - check that the last line in assp.cfg is 'ConfigSavedOK:=1' >> >> - start assp at commandline >> - wait 10 minutes , watch the log for normal maintenance >> - stop assp >> - start it as service >> - wait 10 minutes >> - start a rebuildspamdb task from the GUI or cmdqueue (in doubt enable >> the debugging for the rebuild) >> - wait until this task is finished >> - stop assp >> >> - start assp from commandline - everythng should work like expected >> - stop assp >> - start the service >> >> Thomas >> >> >> Von:"K Post" >> An:"ASSP development mailing list" < >> assp-test@lists.sourceforge.net> >> Datum:29.09.2021 16:41 >> Betreff:[Assp-test] Rebuild SpamDB worker thread at startup hangs >> -- >> >> >> >> If I restart my server or just the ASSP windows service, I've discovered >> that ASSP generally will no longer fully start automatically as a service. >> I'm sure there's something misconfigured, corrupt, or broken on my >> machine. If the Windows server crashes or does an after hours update >> restart, ASSP needs to be started from the command line before it'll run >> as a service, which presents a big problem. >> >> *The service does always run, but I cannot connect to the GUI nor to >> SMTP.* At start, it uses about 10% of processor as expected. In task >> manager, I see memory usage go ~300mb, ~600mb, then up to 984mb and stop. >> CPU then goes down to 0.0% or 0.1% and it just hangs. In testing, I've >> left it for over an hour like this. It just sits. No SMTP, no GUI. RAM >> usage stays at 984mb. Nothing in maillog, nothing in the Windows >> system log that I can find. >> >> If I start from command line, ASSP will work, but there is a delay on the: >> starting rebuild SpamDB worker thread >> line, memory sits at 984mb, then after a two minutes or so, memory grows >> a bit more, I'll get the [OK] on the starting rebuild SpamDB line, and SMTP >> and the GUI *start to work*. It'll run like this quite happily >> forever. >> >> I want it running as a service though, so I'll ctrl+c, then start ASSP >> from the services manager. Task manager will show it again pausing at >> 984mb, but this time, it always continues and operates normally! So the >> temporary fix is to start on command line, ctrl+c, then start the service >> normally. >> >> Everything is up to date, including all modules. I'm using MySQL. >> useDB4Rebuild is checked >> RebuildEsesFileModel is unchecked, though I did try this when it first >> became an option. I turned it off because it used so much RAM and my >> rebuild is already under 45 minutes. Could there be something lingering >> because of this? >> >> Can you suggest anything that I should be checking? >> Is there something that starting ASSP from command line could be clearing >> out or resolving that starting it as a service wouldn't? >> Is there a debug switch that would capture what's happening at this very >> early stage of startup? There doesn't seem to be anything written to the >> logs at all. >> >> I'm making do as is, but sure would like to fix this issue. >> >> Thank you. >> >> >> ___ >> Assp-test mailing list >> Assp-test@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/assp-test >> >> >> >> >> DISCLAIMER: >>
Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs
Thank you Thomas!! I will run through these steps this weekend and report back. On Wed, Sep 29, 2021 at 12:03 PM Thomas Eckardt wrote: > - check all the setting for the lists and hashes to be set to 'DB:' (using > mysql) > - check that the mysql server is working correctly (if not assp will > failover to plain files) > - check all settings related to the rebuildspamdb > - shutdown assp > - clear the folder 'tmpDB' > - remove all 'griplist' related files from assp root folder > - check the file time of the assp.cfg (written at assp shutdown) > - check that the last line in assp.cfg is 'ConfigSavedOK:=1' > > - start assp at commandline > - wait 10 minutes , watch the log for normal maintenance > - stop assp > - start it as service > - wait 10 minutes > - start a rebuildspamdb task from the GUI or cmdqueue (in doubt enable the > debugging for the rebuild) > - wait until this task is finished > - stop assp > > - start assp from commandline - everythng should work like expected > - stop assp > - start the service > > Thomas > > > Von:"K Post" > An:"ASSP development mailing list" < > assp-test@lists.sourceforge.net> > Datum:29.09.2021 16:41 > Betreff:[Assp-test] Rebuild SpamDB worker thread at startup hangs > -- > > > > If I restart my server or just the ASSP windows service, I've discovered > that ASSP generally will no longer fully start automatically as a service. > I'm sure there's something misconfigured, corrupt, or broken on my > machine. If the Windows server crashes or does an after hours update > restart, ASSP needs to be started from the command line before it'll run > as a service, which presents a big problem. > > *The service does always run, but I cannot connect to the GUI nor to SMTP.* > At start, it uses about 10% of processor as expected. In task manager, I > see memory usage go ~300mb, ~600mb, then up to 984mb and stop. CPU then > goes down to 0.0% or 0.1% and it just hangs. In testing, I've left it for > over an hour like this. It just sits. No SMTP, no GUI. RAM usage stays > at 984mb. Nothing in maillog, nothing in the Windows system log that I can > find. > > If I start from command line, ASSP will work, but there is a delay on the: > starting rebuild SpamDB worker thread > line, memory sits at 984mb, then after a two minutes or so, memory grows a > bit more, I'll get the [OK] on the starting rebuild SpamDB line, and SMTP > and the GUI *start to work*. It'll run like this quite happily forever. > > I want it running as a service though, so I'll ctrl+c, then start ASSP > from the services manager. Task manager will show it again pausing at > 984mb, but this time, it always continues and operates normally! So the > temporary fix is to start on command line, ctrl+c, then start the service > normally. > > Everything is up to date, including all modules. I'm using MySQL. > useDB4Rebuild is checked > RebuildEsesFileModel is unchecked, though I did try this when it first > became an option. I turned it off because it used so much RAM and my > rebuild is already under 45 minutes. Could there be something lingering > because of this? > > Can you suggest anything that I should be checking? > Is there something that starting ASSP from command line could be clearing > out or resolving that starting it as a service wouldn't? > Is there a debug switch that would capture what's happening at this very > early stage of startup? There doesn't seem to be anything written to the > logs at all. > > I'm making do as is, but sure would like to fix this issue. > > Thank you. > > > ___ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > DISCLAIMER: > *** > This email and any files transmitted with it may be confidential, legally > privileged and protected in law and are intended solely for the use of the > individual to whom it is addressed. > This email was multiple times scanned for viruses. There should be no > known virus in this email! > *** > > ___ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Rebuild SpamDB worker thread at startup hangs
- check all the setting for the lists and hashes to be set to 'DB:' (using mysql) - check that the mysql server is working correctly (if not assp will failover to plain files) - check all settings related to the rebuildspamdb - shutdown assp - clear the folder 'tmpDB' - remove all 'griplist' related files from assp root folder - check the file time of the assp.cfg (written at assp shutdown) - check that the last line in assp.cfg is 'ConfigSavedOK:=1' - start assp at commandline - wait 10 minutes , watch the log for normal maintenance - stop assp - start it as service - wait 10 minutes - start a rebuildspamdb task from the GUI or cmdqueue (in doubt enable the debugging for the rebuild) - wait until this task is finished - stop assp - start assp from commandline - everythng should work like expected - stop assp - start the service Thomas Von:"K Post" An: "ASSP development mailing list" Datum: 29.09.2021 16:41 Betreff: [Assp-test] Rebuild SpamDB worker thread at startup hangs If I restart my server or just the ASSP windows service, I've discovered that ASSP generally will no longer fully start automatically as a service. I'm sure there's something misconfigured, corrupt, or broken on my machine. If the Windows server crashes or does an after hours update restart, ASSP needs to be started from the command line before it'll run as a service, which presents a big problem. The service does always run, but I cannot connect to the GUI nor to SMTP. At start, it uses about 10% of processor as expected. In task manager, I see memory usage go ~300mb, ~600mb, then up to 984mb and stop. CPU then goes down to 0.0% or 0.1% and it just hangs. In testing, I've left it for over an hour like this. It just sits. No SMTP, no GUI. RAM usage stays at 984mb. Nothing in maillog, nothing in the Windows system log that I can find. If I start from command line, ASSP will work, but there is a delay on the: starting rebuild SpamDB worker thread line, memory sits at 984mb, then after a two minutes or so, memory grows a bit more, I'll get the [OK] on the starting rebuild SpamDB line, and SMTP and the GUI start to work. It'll run like this quite happily forever. I want it running as a service though, so I'll ctrl+c, then start ASSP from the services manager. Task manager will show it again pausing at 984mb, but this time, it always continues and operates normally! So the temporary fix is to start on command line, ctrl+c, then start the service normally. Everything is up to date, including all modules. I'm using MySQL. useDB4Rebuild is checked RebuildEsesFileModel is unchecked, though I did try this when it first became an option. I turned it off because it used so much RAM and my rebuild is already under 45 minutes. Could there be something lingering because of this? Can you suggest anything that I should be checking? Is there something that starting ASSP from command line could be clearing out or resolving that starting it as a service wouldn't? Is there a debug switch that would capture what's happening at this very early stage of startup? There doesn't seem to be anything written to the logs at all. I'm making do as is, but sure would like to fix this issue. Thank you. ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: *** This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *** ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] Rebuild SpamDB worker thread at startup hangs
If I restart my server or just the ASSP windows service, I've discovered that ASSP generally will no longer fully start automatically as a service. I'm sure there's something misconfigured, corrupt, or broken on my machine. If the Windows server crashes or does an after hours update restart, ASSP needs to be started from the command line before it'll run as a service, which presents a big problem. *The service does always run, but I cannot connect to the GUI nor to SMTP.* At start, it uses about 10% of processor as expected. In task manager, I see memory usage go ~300mb, ~600mb, then up to 984mb and stop. CPU then goes down to 0.0% or 0.1% and it just hangs. In testing, I've left it for over an hour like this. It just sits. No SMTP, no GUI. RAM usage stays at 984mb. Nothing in maillog, nothing in the Windows system log that I can find. If I start from command line, ASSP will work, but there is a delay on the: starting rebuild SpamDB worker thread line, memory sits at 984mb, then after a two minutes or so, memory grows a bit more, I'll get the [OK] on the starting rebuild SpamDB line, and SMTP and the GUI *start to work*. It'll run like this quite happily forever. I want it running as a service though, so I'll ctrl+c, then start ASSP from the services manager. Task manager will show it again pausing at 984mb, but this time, it always continues and operates normally! So the temporary fix is to start on command line, ctrl+c, then start the service normally. Everything is up to date, including all modules. I'm using MySQL. useDB4Rebuild is checked RebuildEsesFileModel is unchecked, though I did try this when it first became an option. I turned it off because it used so much RAM and my rebuild is already under 45 minutes. Could there be something lingering because of this? Can you suggest anything that I should be checking? Is there something that starting ASSP from command line could be clearing out or resolving that starting it as a service wouldn't? Is there a debug switch that would capture what's happening at this very early stage of startup? There doesn't seem to be anything written to the logs at all. I'm making do as is, but sure would like to fix this issue. Thank you. ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] rebuild spamdb not cleaning HELO list
Anyone have any ideas? The HELO Blacklist seems to grow each day... tried googling for the issue and nothing similar comes up. However looking at other peoples posted logs of rebuildspamdb it appears that the HELO blacklist should be cleaned each run of rebuildspamdb. -- View this message in context: http://anti-spam-smtp-proxy-server.996265.n3.nabble.com/rebuild-spamdb-not-cleaning-HELO-list-tp37321p37336.html Sent from the assp-test mailing list archive at Nabble.com. -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911iu=/4140/ostg.clktrk ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] Rebuild Spamdb no longer running automatically
Previously the Rebuild Spamdb function would run automatically. This is no longer happening for me. It works fine if I manually run it using the GUI RunRebuildNow function. It's the scheduled run which isn't happening. newReportedInterval is set to '10 5' DoRebuildSpamdb is set to '1' REbuildSchedule is set to 'noschedule' autoCorrectCorpus is '0.99-1.01-4000-14' Running 2.3.4(13171) with Perl 5.012003. Thanks, James. -- This SF.net email is sponsored by Windows: Build for Windows Store. http://p.sf.net/sfu/windows-dev2dev ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] rebuild spamdb - runtime
Hi all, investigating an IMHO too long rebuild runtime issue - I assumed a bug in the code - I've found that this issue was caused by only a few files in the corpus. For some of the files it has taken one minute or more to finish processing them. This large file processing time not depends on the size of the files, it only depends on the contents. All of the files are extremly HTML encoded and contains a very very large number of HTML tags. For example: I removed 6 files from the corpus and the rebuild time is decreased from 20 to 10 minutes. Such a very long processing time could lead in to long time record locks for Spamdb, Whitelistdb and Redlistdb - which could cause any other thread to run in to a stuck mode (get locked) and it will peak the CPU usage for a single core to 100% for a long time. The next relase will monitor the processing time for every file. The list of the longest 10 runtimes will be written to the rebuild log for every corpus folder, if the processing time for a single file is larger than one second. The file 'rebuilddebug.txt' will contain the complete list of files with a processing time 1 second. Thomas DISCLAIMER: *** This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *** -- This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] rebuild spamdb - runtime
How about an option just to delete files in the corpus that cause excessive scanning times? That would be a lot more admin-friendly. At 08:54 AM 3/26/2012, Thomas Eckardt wrote: Hi all, investigating an IMHO too long rebuild runtime issue - I assumed a bug in the code - I've found that this issue was caused by only a few files in the corpus. For some of the files it has taken one minute or more to finish processing them. This large file processing time not depends on the size of the files, it only depends on the contents. All of the files are extremly HTML encoded and contains a very very large number of HTML tags. For example: I removed 6 files from the corpus and the rebuild time is decreased from 20 to 10 minutes. Such a very long processing time could lead in to long time record locks for Spamdb, Whitelistdb and Redlistdb - which could cause any other thread to run in to a stuck mode (get locked) and it will peak the CPU usage for a single core to 100% for a long time. The next relase will monitor the processing time for every file. The list of the longest 10 runtimes will be written to the rebuild log for every corpus folder, if the processing time for a single file is larger than one second. The file 'rebuilddebug.txt' will contain the complete list of files with a processing time 1 second. Thomas DISCLAIMER: *** This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! *** -- This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test -- This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] rebuild spamdb algoritm . . . ASSP v2.1.2 (12067) _2012-03-07 09-37 Re: fixes in assp 2.1.2 build 12067
Hi! See - Original Message - From: Victor Miasnikov To: Michael M Galapchuk ; Scott MacLean Cc: Thomas Eckardt ; Grayhat Sent: Wednesday, March 07, 2012 5:57 PM Subject: Fw: rebuild spamdb algoritm . . . ASSP v2.1.2 (12067) _2012-03-07 09-37 Try assp.pl from attach and/or . . . -- Best regards, Victor Miasnikov Blog: http://vvm.blog.tut.by/ P.S. I'm ( and ex.-USSR people ) go to holiday/celebration 2012-03-08 ( holiday 2012-03-08 - 2012-03-10) - Original Message - From: Michael M Galapchuk michael.galapc...@fcbank.com.ua To: Thomas Eckardt thomas.ecka...@thockar.com Cc: ASSP development mailing list assp-test@lists.sourceforge.net Sent: Wednesday, March 07, 2012 4:57 PM Subject: Re: [Assp-test] fixes in assp 2.1.2 build 12067 Hi Thomas, fixed in assp 2.1.2 build 12067: . . . - reduces the time used for the 'Generating weighted Bayesian tuplets' rebuild task if 'ReplaceOldSpamdb' is set and a DB is used for spamdb I confirm that perl process is crashed by Rebuildspamdb version 3.31 started in ASSP version 2.1.2(12067) even for first 100-200 mails in errors/spam. Mike. -- Virtualization Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] Rebuild spamdb failing
My spamdb rebuild is failing, logging the following - where it shows [null] it is actually inserting null characters into the log. Any idea where it might be picking up these null values from? Mar-05-12 03:34:42 [Worker_10001] Error: rebuildspamdb failed - Database select statement failed: Can't Execute Command 'select pkey from spamdb where pkey='[null] | [null]' ' Package: DBD::ADO::st Filename : C:/Perl/site/lib/DBD/ADO.pm Line : 1244 Last error : -2147217900 OLE exception from Microsoft OLE DB Provider for ODBC Drivers: [Microsoft][SQL Native Client][SQL Server]Executing SQL directly; no cursor. Win32::OLE(0.1709) error 0x80040e14 in METHOD/PROPERTYGET Execute Description : [Microsoft][SQL Native Client][SQL Server]Executing SQL directly; no cursor. HelpContext : HelpFile : NativeError : 16954 Number : -2147217900 Source : Microsoft OLE DB Provider for ODBC Drivers SQLState : 01000 Description : [Microsoft][SQL Native Client][SQL Server]Unclosed quotation mark after the character string ''. HelpContext : HelpFile : NativeError : 105 Number : -2147217900 Source : Microsoft OLE DB Provider for ODBC Drivers SQLState : 37000 Description : [Microsoft][SQL Native Client][SQL Server]Incorrect syntax near ''. HelpContext : HelpFile : NativeError : 102 Number : -2147217900 Source : Microsoft OLE DB Provider for ODBC Drivers SQLState : 37000 Description : [Microsoft][SQL Native Client][SQL Server]Statement(s) could not be prepared. HelpContext : HelpFile : NativeError : 8180 Number : -2147217900 Source : Microsoft OLE DB Provider for ODBC Drivers SQLState : 37000 at D:/ASSP/lib/rebuildspamdb.pm line 572 thread 2. -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] Rebuild SpamDB
I've noticed in the last few 2.x releases that rebuildspamdb is no longer rebuilding every hour, like I've selected. It seems to be running ever 5 or so hours. I'm current running 2.1.2(11316) Doug -- Ben Franklin quote: Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety. -- RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1 ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] Rebuild spamdb questions (Multithreading, uploading GrIP list, rebuild at start)
Hi all! 1. I have about 12k per spam and nospam folders for spamdb and noticed that processing these files is too long in time - about 20 minutes (I think - it's too long, maybe I wrong) Diggin-in-depth said that bottleneck is slow CPU. Meanwile, I have 2 HT enabled CPU's (total is 4 logical CPU) and while spamdb is rebuilds, only one logical CPU is used for this process. Average disk queue produced by PERL.EXE - is 0.10-0.15 and whole system average disk queue (read and write) is 0.23-0.30 (0.50 in peaks). If there is no dependencies in processing of these folders - may it be better to run processing in separate threads with minimal (subject to change in configuration) priority? Also, to prevent high disk load (in case if need to control disk utilization), in configuration neet to be added delay between parsing next file while rebuilding (100 ms for idle priority, 10 - for high speed rebuild). This is lowest priority feature request. 2. Process of uploading GrIP list is TOO slow. In my case it can take about 5-7 minutes. And I don't know where to dig in. There is no high upload internet traffic - I have about 60% free of bandwidth of 2 mbps most of time. Server have direct connect to internet through firewall. Submitted 22548 bytes: 5634 IPv4 addresses - I've think it is NOT a lot of data... Please, advice! 3. And the simpliest question. Please add a checkbox Rebuild spamdb at start. I'll use this because I still have a problem of accident restart of ASSP. Perl process is growing in Working set memory usage and crash. Separate topic will be created after I'll get more info. So I have to use scheduled restart in each 8 hours to prevent crash and instability in last 10-20 minutes before, in which some mail aren't go through ASSP. In testing environment I can't clone this behaviour because I can't do a high load mail traffic. Windows ASSP users (admins, of course :-) ) - please tell me, have you wached such behaviour on your systems? --- File rebuildrun.txt follows: [skipped] c:/assp/spam File Count: 11,254 Processing... spam with 11254 files Imported Files: 11,254 Finished in 494 second(s) c:/assp/notspam File Count: 10,933 Processing... notspam with 10933 files Imported Files: 10,933 Finished in 540 second(s) [skipped] Total processing time: 1227 second(s) Best regards, Alexander Shabalin -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Rebuild spamdb questions (Multithreading, uploading GrIP list, rebuild at start)
Hi Alexander, 1. My spamdb rebuild is taken about the same time (virtual Linux machine, 512mb ram, mysql db for spamdb) for a similar amount of messages, so I think the time taken might be normal. And as far as I know, spamdb rebuild is in a separate thread (worker 1001). 2. I would suggest installing windows netmon or maybe wireshark to monitor the traffic for the grplist upload. 3. I also saw a problem with perl/windows after an extended period of time (but mine was more like 72 hours). System would gradually grind to a halt with perl at 100% process utilization. (it was early 2.x...) it's the main reason I switched to rubbing ASSP on Linux. As far as the test enviroment goes, you could duplicate (90%) by telling your receiving MTA to mirror all traffic to your test ASSP system. That would generate the traffic (but of course, all from 1 ip). Hope this helps Steve Sent from my iPhone On 13.09.2009, at 13:18, Shabalin, Alexander ashaba...@jgl.ru wrote: Hi all! 1. I have about 12k per spam and nospam folders for spamdb and noticed that processing these files is too long in time - about 20 minutes (I think - it's too long, maybe I wrong) Diggin-in-depth said that bottleneck is slow CPU. Meanwile, I have 2 HT enabled CPU's (total is 4 logical CPU) and while spamdb is rebuilds, only one logical CPU is used for this process. Average disk queue produced by PERL.EXE - is 0.10-0.15 and whole system average disk queue (read and write) is 0.23-0.30 (0.50 in peaks). If there is no dependencies in processing of these folders - may it be better to run processing in separate threads with minimal (subject to change in configuration) priority? Also, to prevent high disk load (in case if need to control disk utilization), in configuration neet to be added delay between parsing next file while rebuilding (100 ms for idle priority, 10 - for high speed rebuild). This is lowest priority feature request. 2. Process of uploading GrIP list is TOO slow. In my case it can take about 5-7 minutes. And I don't know where to dig in. There is no high upload internet traffic - I have about 60% free of bandwidth of 2 mbps most of time. Server have direct connect to internet through firewall. Submitted 22548 bytes: 5634 IPv4 addresses - I've think it is NOT a lot of data... Please, advice! 3. And the simpliest question. Please add a checkbox Rebuild spamdb at start. I'll use this because I still have a problem of accident restart of ASSP. Perl process is growing in Working set memory usage and crash. Separate topic will be created after I'll get more info. So I have to use scheduled restart in each 8 hours to prevent crash and instability in last 10-20 minutes before, in which some mail aren't go through ASSP. In testing environment I can't clone this behaviour because I can't do a high load mail traffic. Windows ASSP users (admins, of course :-) ) - please tell me, have you wached such behaviour on your systems? --- File rebuildrun.txt follows: [skipped] c:/assp/spam File Count: 11,254 Processing... spam with 11254 files Imported Files: 11,254 Finished in 494 second(s) c:/assp/notspam File Count: 10,933 Processing... notspam with 10933 files Imported Files: 10,933 Finished in 540 second(s) [skipped] Total processing time: 1227 second(s) Best regards, Alexander Shabalin --- --- --- - Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Rebuild spamdb questions (Multithreading, uploading GrIP list, rebuild at start)
Thanks for good ideas, Steve. I amlost forget about mailflow forwarding. I'll try this thing to simulate mail traffic to testing environment. And yes, spamdb is rebuilt in separate thread (as you rightfully said - worker 1001), but I suggest to separate rebuild process itself into few small pieces, and create one worker for each to do: 1. Parsing Spam folder 2. Parsing NonSpam folder 3. Parsing Errors\Spam 4. Parsing Errors\NotSpam 5. Log parsing imVho - these steps is completely independent from each other, and results were summarized after everyone steps is completed (corpus computing). Best regards, Alexander Shabalin -Original Message- From: Маллиндайн Стивен (Steve Mallindine) [mailto:st...@sc.ru.ru] Sent: 13 сентября 2009 г. 13:58 To: ASSP development mailing list Subject: Re: [Assp-test] Rebuild spamdb questions (Multithreading, uploading GrIP list, rebuild at start) Hi Alexander, 1. My spamdb rebuild is taken about the same time (virtual Linux machine, 512mb ram, mysql db for spamdb) for a similar amount of messages, so I think the time taken might be normal. And as far as I know, spamdb rebuild is in a separate thread (worker 1001). 2. I would suggest installing windows netmon or maybe wireshark to monitor the traffic for the grplist upload. 3. I also saw a problem with perl/windows after an extended period of time (but mine was more like 72 hours). System would gradually grind to a halt with perl at 100% process utilization. (it was early 2.x...) it's the main reason I switched to rubbing ASSP on Linux. As far as the test enviroment goes, you could duplicate (90%) by telling your receiving MTA to mirror all traffic to your test ASSP system. That would generate the traffic (but of course, all from 1 ip). Hope this helps Steve Sent from my iPhone On 13.09.2009, at 13:18, Shabalin, Alexander ashaba...@jgl.ru wrote: Hi all! 1. I have about 12k per spam and nospam folders for spamdb and noticed that processing these files is too long in time - about 20 minutes (I think - it's too long, maybe I wrong) Diggin-in-depth said that bottleneck is slow CPU. Meanwile, I have 2 HT enabled CPU's (total is 4 logical CPU) and while spamdb is rebuilds, only one logical CPU is used for this process. Average disk queue produced by PERL.EXE - is 0.10-0.15 and whole system average disk queue (read and write) is 0.23-0.30 (0.50 in peaks). If there is no dependencies in processing of these folders - may it be better to run processing in separate threads with minimal (subject to change in configuration) priority? Also, to prevent high disk load (in case if need to control disk utilization), in configuration neet to be added delay between parsing next file while rebuilding (100 ms for idle priority, 10 - for high speed rebuild). This is lowest priority feature request. 2. Process of uploading GrIP list is TOO slow. In my case it can take about 5-7 minutes. And I don't know where to dig in. There is no high upload internet traffic - I have about 60% free of bandwidth of 2 mbps most of time. Server have direct connect to internet through firewall. Submitted 22548 bytes: 5634 IPv4 addresses - I've think it is NOT a lot of data... Please, advice! 3. And the simpliest question. Please add a checkbox Rebuild spamdb at start. I'll use this because I still have a problem of accident restart of ASSP. Perl process is growing in Working set memory usage and crash. Separate topic will be created after I'll get more info. So I have to use scheduled restart in each 8 hours to prevent crash and instability in last 10-20 minutes before, in which some mail aren't go through ASSP. In testing environment I can't clone this behaviour because I can't do a high load mail traffic. Windows ASSP users (admins, of course :-) ) - please tell me, have you wached such behaviour on your systems? --- File rebuildrun.txt follows: [skipped] c:/assp/spam File Count: 11,254 Processing... spam with 11254 files Imported Files: 11,254 Finished in 494 second(s) c:/assp/notspam File Count: 10,933 Processing... notspam with 10933 files Imported Files: 10,933 Finished in 540 second(s) [skipped] Total processing time: 1227 second(s) Best regards, Alexander Shabalin -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] Rebuild Spamdb
I know I might be the only one with this concern but thought I would put it out there any way. I like that the process is now part of the assp application but only having the ability to schedule it to run on Interval in hours is sort of a problem in my case as it loads up the system at in opportune times. For me anyhow having the ability to say set it to run at midnight or at time deem low peak hours would be helpful. Just a thought, Daniel Du Vall - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Rebuild Spamdb
Daniel K. Du Vall wrote: I know I might be the only one with this concern but thought I would put it out there any way. I like that the process is now part of the assp application but only having the ability to schedule it to run on Interval in hours is sort of a problem in my case as it loads up the system at in opportune times. For me anyhow having the ability to say set it to run at midnight or at time deem low peak hours would be helpful. Just a thought, Daniel Du Vall Would it be possible to have a learning mode, so ASSP auto-detects an optimal run-schedule for the rebuild process? Based on both the frequency of spam/ham updates and peak usage? -- Daniel - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
