Thanks for good ideas, Steve. I amlost forget about mailflow forwarding. I'll try this thing to simulate mail traffic to testing environment.
And yes, spamdb is rebuilt in separate thread (as you rightfully said - worker 1001), but I suggest to separate rebuild process itself into few small pieces, and create one worker for each to do: 1. Parsing Spam folder 2. Parsing NonSpam folder 3. Parsing Errors\Spam 4. Parsing Errors\NotSpam 5. Log parsing imVho - these steps is completely independent from each other, and results were summarized after everyone steps is completed (corpus computing). Best regards, Alexander Shabalin -----Original Message----- From: Маллиндайн Стивен (Steve Mallindine) [mailto:st...@sc.ru.ru] Sent: 13 сентября 2009 г. 13:58 To: ASSP development mailing list Subject: Re: [Assp-test] Rebuild spamdb questions (Multithreading, uploading GrIP list, rebuild at start) Hi Alexander, 1. My spamdb rebuild is taken about the same time (virtual Linux machine, 512mb ram, mysql db for spamdb) for a similar amount of messages, so I think the time taken might be normal. And as far as I know, spamdb rebuild is in a separate thread (worker 1001). 2. I would suggest installing windows netmon or maybe wireshark to monitor the traffic for the grplist upload. 3. I also saw a problem with perl/windows after an extended period of time (but mine was more like 72 hours). System would gradually grind to a halt with perl at 100% process utilization. (it was early 2.x...) it's the main reason I switched to rubbing ASSP on Linux. As far as the test enviroment goes, you could duplicate (90%) by telling your receiving MTA to mirror all traffic to your test ASSP system. That would generate the traffic (but of course, all from 1 ip). Hope this helps Steve Sent from my iPhone On 13.09.2009, at 13:18, "Shabalin, Alexander" <ashaba...@jgl.ru> wrote: > Hi all! > > 1. > I have about 12k per spam and nospam folders for spamdb and noticed > that > processing these files is too long in time - about 20 minutes (I > think - > it's too long, maybe I wrong) > Diggin-in-depth said that bottleneck is slow CPU. Meanwile, I have 2 > HT > enabled CPU's (total is 4 logical CPU) and while spamdb is rebuilds, > only one logical CPU is used for this process. > Average disk queue produced by PERL.EXE - is 0.10-0.15 and whole > system > average disk queue (read and write) is 0.23-0.30 (0.50 in peaks). > > If there is no dependencies in processing of these folders - may it be > better to run processing in separate threads with minimal (subject to > change in configuration) priority? > Also, to prevent high disk load (in case if need to control disk > utilization), in configuration neet to be added delay between parsing > next file while rebuilding (100 ms for idle priority, 10 - for high > speed rebuild). > > This is lowest priority feature request. > > 2. > Process of uploading GrIP list is TOO slow. In my case it can take > about > 5-7 minutes. And I don't know where to dig in. > There is no high upload internet traffic - I have about 60% free of > bandwidth of 2 mbps most of time. > Server have direct connect to internet through firewall. > Submitted 22548 bytes: 5634 IPv4 addresses - I've think it is NOT a > lot > of data... > Please, advice! > > 3. > And the simpliest question. > Please add a checkbox "Rebuild spamdb at start". > I'll use this because I still have a problem of accident restart of > ASSP. Perl process is growing in Working set memory usage and crash. > Separate topic will be created after I'll get more info. > So I have to use scheduled restart in each 8 hours to prevent crash > and > instability in last 10-20 minutes before, in which some mail aren't go > through ASSP. > In testing environment I can't clone this behaviour because I can't > do a > high load mail traffic. > Windows ASSP users (admins, of course :-) ) - please tell me, have you > wached such behaviour on your systems? > > --------------- > File rebuildrun.txt follows: > > [skipped] > > c:/assp/spam > File Count: 11,254 > Processing... spam with 11254 files > Imported Files: 11,254 > Finished in 494 second(s) > > c:/assp/notspam > File Count: 10,933 > Processing... notspam with 10933 files > Imported Files: 10,933 > Finished in 540 second(s) > > [skipped] > > Total processing time: 1227 second(s) > > > Best regards, > Alexander Shabalin ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
