[Assp-test] Antwort: Re: Speeding up MySQL
forgot my speed results Sep-13-12 04:15:43 Generating weighted Bayesian tuplets Sep-13-12 04:15:59 start populating Spamdb with 172,557 records - Bayesian check is now disabled! Sep-13-12 04:16:43 Finished populating Spamdb with 172,557 records - Bayesian check is now enabled! 44 seconds for 172.557 spamdb records (include a fixed delay of 5 seconds - so it takes really 39 seconds) Sep-13-12 04:18:44 start populating Hidden Markov Model with 774,061 records! Sep-13-12 04:20:43 Finished populating Hidden Markov Model with 774,061 records! 2 miniutes (119 seconds) for 774.061 hmmdb records The performance would be better if this system were not a allrounder. In german we say an oviparous wolled sow that gives milk. Thomas Von:Thomas Eckardt thomas.ecka...@thockar.com An: ASSP development mailing list assp-test@lists.sourceforge.net, Datum: 13.09.2012 07:52 Betreff:[Assp-test] Antwort: Re: Speeding up MySQL SELECT COUNT(*) FROM hmmdb takes 22 seconds. Looks like you don't use INNODB for the tables or your INNODB cache is too small. hmm use 100MB and apmdb 15MB on my system the relevant settings for mysql and INNODB are max_allowed_packet=128M net_buffer_length=32M sql-mode=STRICT_ALL_TABLES,STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION max_connections=800 query_cache_size=84M table_cache=1520 tmp_table_size=30M innodb_additional_mem_pool_size=6M innodb_flush_log_at_trx_commit=1 innodb_log_buffer_size=30M innodb_buffer_pool_size=250M innodb_log_file_size=512M innodb_thread_concurrency=8 innodb_file_io_threads=4 Thomas Von:Colin a...@lanternhosting.co.uk An: assp-test@lists.sourceforge.net, Datum: 13.09.2012 01:41 Betreff:Re: [Assp-test] Speeding up MySQL Thanks for the tips, I have turned on the slow query logging and it only shows two queries. DELETE FROM hmmdb takes just under two minutes and SELECT COUNT(*) FROM hmmdb takes 22 seconds. Nothing else is logged. I'll investigate mysqltuner further tomorrow - it thinks that mysql is requiring 13GB at its biggest which is rather big! I'm not sure fragmentation will make much of a difference as the HMM db is emptied before it is repopulated Thanks for the suggestions, I've never really run any optimisation on mySQL so any tips are appreciated. All the best, Colin Waring. On 12/09/2012 22:15, Rusty Nejdl wrote: Here's some generic mysql tuning info: Turn on logging of slow queries and figure out which queries are taking a while. Run them through an analyzer to see if any indexes are missing. mysqltuner and mysqlfragfinder are two scripts you can run to help tune the database. (If I am telling you stuff you already know, don't take it personally.) Rusty Nejdl On 2012-09-12 15:37, Colin wrote: I think this came up a long time ago, but the populating Hidden Markov Model part of rebuildspamdb takes a very long time on my system (over an hour). This time I have been keeping an eye on it and I have no idea why. For the majority of that hour, there is no CPU activity, no network activity and a very small amount of disk write activity. For the last minute, there was a burst of network and CPU activity. Has anyone experience with how to tune up mysql so it works a bit quicker during the rebuild but doesn't negatively affect the regular operations during the day? This server does nothing except the ASSP database.. All the best, Colin Waring. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats.
Re: [Assp-test] Antwort: Re: Speeding up MySQL
I think that might be it, Innodb doesn't appear to be enabled by default in the install so I have converted everything over and fine tuned some of those settings with recommendations from mysqltuner as well. We shall see how it goes at the next rebuild run. Thanks. Colin. On 13/09/2012 06:50, Thomas Eckardt wrote: SELECT COUNT(*) FROM hmmdb takes 22 seconds. Looks like you don't use INNODB for the tables or your INNODB cache is too small. hmm use 100MB and apmdb 15MB on my system the relevant settings for mysql and INNODB are max_allowed_packet=128M net_buffer_length=32M sql-mode=STRICT_ALL_TABLES,STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION max_connections=800 query_cache_size=84M table_cache=1520 tmp_table_size=30M innodb_additional_mem_pool_size=6M innodb_flush_log_at_trx_commit=1 innodb_log_buffer_size=30M innodb_buffer_pool_size=250M innodb_log_file_size=512M innodb_thread_concurrency=8 innodb_file_io_threads=4 Thomas Von:Colin a...@lanternhosting.co.uk An: assp-test@lists.sourceforge.net, Datum: 13.09.2012 01:41 Betreff:Re: [Assp-test] Speeding up MySQL Thanks for the tips, I have turned on the slow query logging and it only shows two queries. DELETE FROM hmmdb takes just under two minutes and SELECT COUNT(*) FROM hmmdb takes 22 seconds. Nothing else is logged. I'll investigate mysqltuner further tomorrow - it thinks that mysql is requiring 13GB at its biggest which is rather big! I'm not sure fragmentation will make much of a difference as the HMM db is emptied before it is repopulated Thanks for the suggestions, I've never really run any optimisation on mySQL so any tips are appreciated. All the best, Colin Waring. On 12/09/2012 22:15, Rusty Nejdl wrote: Here's some generic mysql tuning info: Turn on logging of slow queries and figure out which queries are taking a while. Run them through an analyzer to see if any indexes are missing. mysqltuner and mysqlfragfinder are two scripts you can run to help tune the database. (If I am telling you stuff you already know, don't take it personally.) Rusty Nejdl On 2012-09-12 15:37, Colin wrote: I think this came up a long time ago, but the populating Hidden Markov Model part of rebuildspamdb takes a very long time on my system (over an hour). This time I have been keeping an eye on it and I have no idea why. For the majority of that hour, there is no CPU activity, no network activity and a very small amount of disk write activity. For the last minute, there was a burst of network and CPU activity. Has anyone experience with how to tune up mysql so it works a bit quicker during the rebuild but doesn't negatively affect the regular operations during the day? This server does nothing except the ASSP database.. All the best, Colin Waring. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test DISCLAIMER: *** This email and any files transmitted with it may be confidential, legally privileged and protected in law and are intended solely for the use of the individual to whom it is addressed. This email was multiple times scanned for viruses. There should be no known virus in this email! ***
[Assp-test] PTR missing and PTR invalid don't score
I have setup assp ( 1.9.3.9 ) to scoring. But it seems that the PTR-Lookup doesn't add the score to the totals: the settings are: ptrTestMode:=1 DoPTRCheck:=3 DoPTRCheckInvalid:=1 invalidPTRRe:=file:files/invalidptr.txt validPTRRe:=static whitePTRRe:= PTRCacheExp:=72 The following parts of the log show, what I mean: Sep-12-12 20:28:23 34747-00367 [PTRmissing] 49.249.55.109 boyce-andersony...@chiprunner.ca to: (mailadress) [scoring:50] (PTR missing); Sep-12-12 20:28:23 34747-00367 [VIRUS] 49.249.55.109 boyce-andersony...@chiprunner.ca to: (mailadress) ClamAV: scanned 1100 bytes in message - OK ; Sep-12-12 20:28:23 34747-00367 49.249.55.109 boyce-andersony...@chiprunner.ca to: (mailadress) Bayesian Check [scoring:45] - Prob: 1.0 = spam; Sep-12-12 20:28:23 34747-00367 49.249.55.109 boyce-andersony...@chiprunner.ca to: (mailadress) Message-Score: added 45 for Bayesian Probability: 1.0, total score for this message is now 45; shouldn't the total score be 95? Sep-12-12 21:38:02 34747-00374 [DNSBL] 109.243.181.242 evelynnmar...@alerton.com to: (mailadress) [scoring:1024] DNSBL: neutral, 109.243.181.242 listed in safe.dnsbl.sorbs.net; Sep-12-12 21:38:02 34747-00374 109.243.181.242 evelynnmar...@alerton.com to: (mailadress) Message-Score: added 1024 for DNSBL: neutral, 109.243.181.242 listed in safe.dnsbl.sorbs.net, total score for this message is now 1073; Sep-12-12 21:38:03 34747-00374 [PTRinvalid] 109.243.181.242 evelynnmar...@alerton.com to: (mailadress) [scoring:50] (PTR invalid 'user-109-243-181-242.play-internet.pl'); Sep-12-12 21:38:03 34747-00374 [VIRUS] 109.243.181.242 evelynnmar...@alerton.com to: (mailadress) ClamAV: scanned 799 bytes in message - OK ; Sep-12-12 21:38:03 34747-00374 [BombData] 109.243.181.242 evelynnmar...@alerton.com to: (mailadress) [scoring:25] -- bombDataRe: ' cialis (25)'; Sep-12-12 21:38:03 34747-00374 109.243.181.242 evelynnmar...@alerton.com to: (mailadress) Message-Score: added 25 for bombDataRe: ' cialis (25)', total score for this message is now 1098; should be 1073+50+25=1148 ?? Mit freundlichen Grüßen i.V. Dieter Centner __ PAZDERA AG IT-Leiter Kronacher Straße 34 95119 Naila Telefon: +49 9282 930-231 Fax: +49 9282 930-111 E-Mail: dcent...@pazdera.de __ Sitz der Gesellschaft: Coburg Handelsregister Coburg HRB 4320 Aufsichtsratsvorsitzender: Holger Köthe Vorstand: Michael Pazdera (Vors.) -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] PTR missing and PTR invalid don't score
ASSP development mailing list assp-test@lists.sourceforge.net schrei bt: But it seems that the PTR-Lookup doesn't add the score to the totals: That is a bug. Thanks. Please try: http://downloads.sourceforge.net/project/assp/ASSP%20Installation/Auto Update/ASSP1dev/assp.pl.gz -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
[Assp-test] Question.....
How can I allow this through? It has just started happening and I email receive them a heck of a lot... The ip domain name are in the whitelist already... Sep-13-12 10:36:45 [Worker_1] [TLS-out] [ForgedHELO] 66.55.121.50 mana...@dickinson.bm (forged Helo: 'dickinson.bm') Sep-13-12 10:36:45 [Worker_1] [TLS-out] 66.55.121.50 mana...@dickinson.bm Message-Score: added 150 (fhValencePBhttps://assp.optimum.bm:82/#fhValencePB) for forged Helo: 'dickinson.bm', total score for this message is now 150 Thanks Steve Steve Moffat Operations Director Optimum IT Solutions Desk: 441 292 8849 Mobile: 441 292 8849 MSN IM: st...@optimum.bmmailto:st...@optimum.bm Web: http://www.optimum.bmhttp://www.optimum.bm/ -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Question.....
ASSP development mailing list assp-test@lists.sourceforge.net schrei bt: Helo: 'dickinson.bm' There is an option called: heloBlacklistIgnore -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Question.....
Thanks...I suppose I should have looked..:), Sorry -Original Message- From: Fritz Borgstedt [mailto:f...@iworld.de] Sent: Thursday, September 13, 2012 12:14 PM To: ASSP development mailing list Subject: Re: [Assp-test] Question. ASSP development mailing list assp-test@lists.sourceforge.net schrei bt: Helo: 'dickinson.bm' There is an option called: heloBlacklistIgnore -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
Re: [Assp-test] Block spoofed addresses
On 8/21/2012 11:52 AM, Daniel L. Miller wrote:
I haven't seen any more of the spoofed efax messages reach my printer -
and I'm seeing SPFerrors, fails, neutrals, softfails, and
softfail-stricts in my scoring statistics. Has anyone else seen the
spoofed efax messages blocked as a result of this?
Ok - garbage efax's coming through again. I thought the new SPF record
would block this crap - but it's coming through again. Here is the
analysis of the message:
*Feature Matching:*
*•NoProcessing Domain
http://bubba.amfeslan.local:5/#noProcessingDomains*: 'efax.com'
*•SPF-check returned OK*for65.217.137.4
javascript:void(0);-mess...@inbound.efax.com javascript:void(0);,
[65.217.137.4 javascript:void(0);]
•SPF: fail (cache) ip=65.217.137.4
javascript:void(0);mailfrom=mess...@inbound.efax.com
javascript:void(0);helo=[65.217.137.4 javascript:void(0);]
*•bombRehttp://bubba.amfeslan.local:5/#bombRe*: 'highest match:
ratee with valence: 30 - PB value = 60'
•matching bombRe(file:files/bombre.txt[line 394] javascript:void(0);):
'ratee'
*•URIBL check http://bubba.amfeslan.local:5/#ValidateURIBL*: 'OK'
*•Not a Valid Format of HELO
http://bubba.amfeslan.local:5/#DoValidFormatHelo*: '[65.217.137.4
javascript:void(0);]'
*•Invalid Format of HELO
http://bubba.amfeslan.local:5/#invalidFormatHeloRe*: 'highest
match: 65.217.137 with valence: 20 - PB value = 20'
•matching invalidFormatHeloRe(file:files/invalidhelo.txt[line 4]
javascript:void(0);): '\d{1,3}[-x.]\d{1,3}[-x.]\d{1,3}'
*•IP in Helo check http://bubba.amfeslan.local:5/#DoIPinHelo*: 'OK'
*•RBLCheck returned OK for67.112.61.71 javascript:void(0);*:
*•RBLCheck returned OK for65.217.137.4 javascript:void(0);*: DNSBL:
neutral,65.217.137.4 javascript:void(0);listed in bb.barracudacentral.org
*•65.217.137.0 javascript:void(0);has a Griplist value of 0.8*
--
Daniel
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
