I think the speed is limited by the DB engine not by the network speed.
Thomas
Von: Colin <a...@lanternhosting.co.uk>
An: assp-test@lists.sourceforge.net
Datum: 25.04.2012 20:10
Betreff: Re: [Assp-test] Antwort: Re: Antwort: Slow processing
rebuild output
2012-04-25 11:56:39 Start populating Hidden Markov Model. HMM-check is
disabled for this time!
2012-04-25 11:56:40 start populating Hidden Markov Model ham chains with
653223 records!
2012-04-25 12:00:33 Finished populating Hidden Markov Model ham chains
with 653223 records!
2012-04-25 12:00:33 start populating Hidden Markov Model ham totals with
591133 records!
2012-04-25 12:03:51 Finished populating Hidden Markov Model ham totals
with 591133 records!
2012-04-25 12:03:52 start populating Hidden Markov Model spam chains
with 1901014 records!
2012-04-25 12:38:54 Finished populating Hidden Markov Model spam chains
with 1901014 records!
2012-04-25 12:38:54 start populating Hidden Markov Model spam totals
with 1748208 records!
2012-04-25 13:03:27 Finished populating Hidden Markov Model spam totals
with 1748208 records!
2012-04-25 13:03:27 Finished populating Hidden Markov Model. HMM-check
is now enabled again!
2012-04-25 11:21:43 /usr/local/assp/store/spam
2012-04-25 11:21:43 File Count: 6,318
2012-04-25 11:21:43 Processing... store/spam with 6318 files
2012-04-25 11:42:09 Imported Files: 6,303
2012-04-25 11:42:09 Finished in 1226 second(s)
The link is currently a 100Mb link. As they are VMs on the same machine
I should really set up a private interface so they don't use the
external network.
Would you expect faster out of 100Mb?
On 25/04/2012 18:17, Thomas Eckardt wrote:
>> An hour of that was the HMM generation
> This is really too long.
>
> You can't see the generation of the HMM in the rebuild log - you can
only
> see the populating time, which is a simple DB import.
> My (more slow than fast) mysql server stores ~ 500.000 - 700.000
records
> per minute.
>
>
> Apr-25-12 04:12:04 start populating Hidden Markov Model spam chains with
> 669292 records!
> Apr-25-12 04:13:12 Finished populating Hidden Markov Model spam chains
> with 669292 records!
> Apr-25-12 04:13:12 start populating Hidden Markov Model spam totals with
> 625498 records!
> Apr-25-12 04:14:19 Finished populating Hidden Markov Model spam totals
> with 625498 records!
>
> The HMM build is done in the same task like bayes.
>
> Apr-25-12 04:04:21 c:/assp/spam
> Apr-25-12 04:04:21 File Count: 1,562
> Apr-25-12 04:04:21 Processing... spam with 1562 files
> Apr-25-12 04:04:23 ignore and remove files older than Apr-05-12 04:04:21
> in folder spam
> Apr-25-12 04:09:16 114 attachment/image entries processed
> Apr-25-12 04:09:16 Imported Files: 1,560
> Apr-25-12 04:09:16 Finished in 295 second(s)
>
> Thomas
>
>
>
> Von: Colin<a...@lanternhosting.co.uk>
> An: assp-test@lists.sourceforge.net
> Datum: 25.04.2012 18:56
> Betreff: Re: [Assp-test] Antwort: Slow processing rebuild output
>
>
>
> Thanks,
>
> After correcting the ramdisk problem my rebuildspamdb went from taking
> 9+ hours to an hour and a half. An hour of that was the HMM generation,
> is that always slower than bayes or do you think there might be anything
> I can do to speed that up a bit more?
>
> I didn't get any slow files reported with a ramdisk, though the
> processing speed only made it up to 7 per second and not the expected
12.
>
> All the best,
> Colin.
>
>
>
> On 25/04/2012 17:45, Thomas Eckardt wrote:
>>> What then follows is a list of the full 490 files, not just the 10
>> longest.
>>
>> this will be fixed - meanwhile increase the first value of
>> 'RebuildFileTimeLimit' .
>>
>> Thomas
>>
>>
>>
>> Von: Colin<a...@lanternhosting.co.uk>
>> An: ASSP development mailing list<assp-test@lists.sourceforge.net>
>> Datum: 25.04.2012 12:52
>> Betreff: [Assp-test] Slow processing rebuild output
>>
>>
>>
>> Hi there,
>>
>> My rebuildspamdb has been taking a very long time lately, I realised
> that
>> I hadn't set up a ramdisk for the tmpDB folder on my secondary since
> doing
>> mysql so I have done that and the difference is amazing. Hopefully
> that's
>> solved it!
>>
>> Because the rebuild has been running slow, I have noticed something
that
>> may have slipped past other people. Each night I would get something
> like
>> this:
>>
>> 2012-04-25 04:21:06 The processing time of 490 file(s) in
>> '/usr/local/assp/store/spam/' was longer than 1 second(s) - now showing
>> the 10 longest
>>
>> What then follows is a list of the full 490 files, not just the 10
>> longest.
>>
>> This has happened every night for all the corpus folders so is the
> script
>> somehow missing the check that makes it only output the 10 longest?
>>
>> Thanks for all the hard work so far!
>>
>> Regards
>> Colin.
>>
>>
>
------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond.
> Discussions
>> will include endpoint security, mobile security and the latest in
> malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential,
> legally
>> privileged and protected in law and are intended solely for the use of
> the
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>>
>>
>>
>
------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond.
> Discussions
>> will include endpoint security, mobile security and the latest in
> malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>
>>
>> _______________________________________________
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>
------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond.
Discussions
> will include endpoint security, mobile security and the latest in
malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential,
legally
> privileged and protected in law and are intended solely for the use of
the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
>
>
>
------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond.
Discussions
> will include endpoint security, mobile security and the latest in
malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
