Re: [Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
Hi there, On Fri, 6 Nov 2009 Nathan Gibbs wrote: Cut these guys some slack. They can't do everything. This isn't about cutting slack, or for that matter pointing fingers. It's about contributing. I'm contributing. Sometimes people don't want to hear what needs to be said. That can be painful all round, believe me. Thats the beauty of Open Source. Let's just say I'm familiar with the concept and leave it at that. We don't want you to get your foot completely stuck. :) ... report that its broken. If possible Where is it broken, why is it borken, and how it might be fixed. That's exactly what I'm doing here. Unfortunately my message is not one which people often want to hear. They want to hear what a great job they do, not how they could improve on how they're doing things by doing things for which they really have little sympathy. There isn't some piece of code somewhere that has to be tweaked. If it were so it would be great, they'd just go off and tweak it and all would be well. But it isn't like that. The issue here is that if there is any system which is supposed to prevent problems of the sort that we see so often in releases of ClamAV, then demonstrably it isn't working. My contention is that there is no system, and that's what needs to be fixed. But as I don't employ the developers, all I can do is point to some of the benefits of using modern (i.e. developed in the past fifty years or so) quality systems. We no longer have to promise that the architect will be executed if his building falls down. But there are still people in governments talking about penalty clauses in software procurement contracts because they don't know any better. Sorry, any better way to get the systems delivered on time, under budget and with acceptable levels of faults and performance. Quality systems can and do result in improvements of a couple of orders of magnitude in fault density in software (measured for example in coding errors per 1000 lines of code). The user's experience will inevitably improve as a result - and let's not forget that one coding error can easily cost several hours of the time of each of thousands of the code's users - but in addition, the developers will be able to get on with their developing instead of spending large chunks of time tracking down the faults which they themselves put there because they didn't have some kind of system. Obviously I don't claim that all errors can be eliminated, but I have seen the results of using and failing to use quality systems and the differences can be stunning. It's no exaggeration to say that the cost savings in industry are phenomenal. I've seen manufacturing cost reduced by more than 60% and reliability improved by 500% as a result of proper quality control procedures being applied. I've also had to fly to thirty of the United States to replace four 10 cent components in a couple of hundred examples of a fifty-thousand dollar product because the quality control procedures were ignored. The cost to one company of ignoring a few simple quality procedures which would have found the problem at the sub-assembly manufacturing stage was tens of thousands of dollars in repair bills and very much more than that in loss of customer goodwill. The cost of finding and fixing the problem if the procedures had been followed would have been a few dollars. This is a natural result of the logarithmic relationship between the cost per fix and the stage in the product's life at which the fix is applied. It doesn't really matter what the product is, and despite what you're thinking right now it is _very_ relevant to software, if the software is built in anything like a modular fashion as probably it should be. Here's an example of an implementation of some of the ideas: http://www.praxis-his.com/news/sparkGPL.asp But you don't have to implement the whole thing as an automated toolset, a lot of it can be paper-based or some other manual method such as just writing in a comment at the top of every function what the expected inputs and outputs are, then making sure (by inspection or by test procedures in the code itself, and preferably both) that those are indeed the things that happen in practice. As I said in my first post, quality systems are fundamentally about documentation. We all know what a popular subject that is. :( One aspect of the documentation in this case would be writing down how to prevent these things from happening. That's the first problem. If you can't do that then you're in real trouble, it probably means that you don't know _why_ the things are happening. But at least now you know what the real problem is. It has little to do with the code. It will take time and a little effort, but once it's done, then all you have to do is what you wrote down you would do. That just takes discipline. Admittedly this is another relatively unpopular concept in the programming community. :) There are very many places to start
Re: [Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
* G.W. Haywood wrote: Hi there, On Fri, 6 Nov 2009 Nathan Gibbs wrote: Cut these guys some slack. They can't do everything. This isn't about cutting slack, or for that matter pointing fingers. It's about contributing. I'm contributing. Sometimes people don't want to hear what needs to be said. That can be painful all round, believe me. I'm starting to see what you mean. Thats the beauty of Open Source. Let's just say I'm familiar with the concept and leave it at that. We don't want you to get your foot completely stuck. :) I appreciate that, it won't be the first time or the last. Agree to disagree ? :-) ... report that its broken. If possible Where is it broken, why is it borken, and how it might be fixed. That's exactly what I'm doing here. Unfortunately my message is not one which people often want to hear. They want to hear what a great job they do, not how they could improve on how they're doing things by doing things for which they really have little sympathy. Right. There isn't some piece of code somewhere that has to be tweaked. If it were so it would be great, they'd just go off and tweak it and all would be well. But it isn't like that. The issue here is that if there is any system which is supposed to prevent problems of the sort that we see so often in releases of ClamAV, then demonstrably it isn't working. I'll admin that the 0.94 and 0.95 releases have been a bit more interesting than just install it and move on. The initial clamav 0.95 upgrade broke some internal code here. The EOL note for 0.94 seems to indicate that after 4-15-2010 anything that isn't upgraded will blow. Personally, ( switching feet here ) I don't fault the Clamav team for these decisions, although I feel it is a little rough on the user base. I feel that the code is improving, and these are just some things we will have to deal with. My contention is that there is no system, I'll disagree with you there. One man's chaos is another mans order. I'm not missing your point that the system could improve. and that's what needs to be fixed. But as I don't employ the developers, all I can do is point to some of the benefits of using modern (i.e. developed in the past fifty years or so) quality systems. Snip Quality systems can and do result in improvements of a couple of orders of magnitude in fault density in software (measured for example in coding errors per 1000 lines of code). The user's experience will inevitably improve as a result - and let's not forget that one coding error can easily cost several hours of the time of each of thousands of the code's users - but in addition, the developers will be able to get on with their developing instead of spending large chunks of time tracking down the faults which they themselves put there because they didn't have some kind of system. Been there, done that, no fun. Obviously I don't claim that all errors can be eliminated, but I have seen the results of using and failing to use quality systems and the differences can be stunning. Snip Agreed. The cost to one company of ignoring a few simple quality procedures ... was ... loss of customer goodwill. Which in open source is one thing you don't want to lose. Snip It doesn't really matter what the product is, and despite what you're thinking right now it is _very_ relevant to software, if the software is built in anything like a modular fashion as probably it should be. Agreed, its far less expensive to fix an error before release than after. But you don't have to implement the whole thing as an automated toolset, a lot of it can be paper-based or some other manual method such as just writing in a comment at the top of every function what the expected inputs and outputs are, then making sure (by inspection or by test procedures in the code itself, and preferably both) that those are indeed the things that happen in practice. Eight. As I said in my first post, quality systems are fundamentally about documentation. We all know what a popular subject that is. :( One aspect of the documentation in this case would be writing down how to prevent these things from happening. That's the first problem. If you can't do that then you're in real trouble, it probably means that you don't know _why_ the things are happening. But at least now you know what the real problem is. It has little to do with the code. It will take time and a little effort, but once it's done, then all you have to do is what you wrote down you would do. That just takes discipline. Admittedly this is another relatively unpopular concept in the programming community. :) You are SO RIGHT on that point. A massive benefit can be obtained by documenting ( even informally ) what you are doing. This precisely hits some of the issues around a recent bug I put into the tracker. Whether they add the feature or not isn't
Re: [Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
* aCaB wrote: G.W. Haywood wrote: I suspect that rather than QA, what you do is just a lot of hap-hazard testing. That's why, whenever I see a new release of ClamAV, first I will suppress a groan and then, before I risk it on any of my servers, I'll wait a while and watch the users' list to see how much trouble it causes. This approach serves me well, although I can't say I'm proud of the fact that I'm letting a lot of poor innocents do my acceptance testing for me. Hi G.W. Haywood, My mail was about custom databases provided by 3rd parties, not about ClamAV release cycles. Besides, you miss another point: ClamAV is an open source software, consisting of roughly 150K lines of C code and 65 signatures, currently maintained by three full time developers, one and a half full time sigmakers and a system administrator. So less than 6 people maintain Clamav. I'm impressed. Cut these guys some slack. They can't do everything. Thats the beauty of Open Source. If you don't like it FIX IT. If you can't fix it, report that its broken. If possible Where is it broken, why is it borken, and how it might be fixed. If you can't do that, Get a commercial solution. Freedom isn't free. ( from effort or errors ) -- Sincerely, Nathan Gibbs Systems Administrator Christ Media http://www.cmpublishers.com signature.asc Description: OpenPGP digital signature ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
Hello again, On Wed, 4 Nov 2009 aCaB wrote: G.W. Haywood wrote: I suspect that rather than QA, what you do is just a lot of hap-hazard testing. My mail was about custom databases provided by 3rd parties, not about ClamAV release cycles. Besides, you miss another point: ClamAV is an open source software, I don't think the points were lost on me. We ALWAYS ask our users to test the development head That's exactly what I said. ... bugzilla being all quiet for weeks, then, as soon as we release a new version, it suddently gets flooded with tickets. That's exactly what I said. So, to conclude, if you want to get better releases, do your bit. That's exactly what I am doing now. The trouble is you aren't in a very receptive frame of mind. I'm not going to do what you want me to do because what you want me to do is more hap-hazard testing, and that's what I'm telling you is almost a complete waste of time. The only alternative is that we release what WE think is ok and we re-release when YOU tell us it's not. Not so. One alternative is to do something like I'm suggesting. Thanks for the lesson, Please leave the sarcasm at home tomorrow. It was YOU who asked for thoughts. Didn't you say something about doing my bit? That's what I'm trying to do now. It's a shame you're not listening when people give you the thoughts that you asked for. I'm telling you that your take on how software should be developed is very seriously lacking. Until you start to listen to that you'll get absolutely nowhere. This really does demand a completely different approach to the way software is built, taking it out of the realms of what Professor Thomas calls 'craft technology' and into the mainstream of engineering. In a word, it's called 'quality'. Most people haven't the faintest idea what it's about. Read about it, and about what it can achieve. You'll be surprised. -- 73, Ged. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
G.W. Haywood wrote: I suspect that rather than QA, what you do is just a lot of hap-hazard testing. That's why, whenever I see a new release of ClamAV, first I will suppress a groan and then, before I risk it on any of my servers, I'll wait a while and watch the users' list to see how much trouble it causes. This approach serves me well, although I can't say I'm proud of the fact that I'm letting a lot of poor innocents do my acceptance testing for me. Hi G.W. Haywood, My mail was about custom databases provided by 3rd parties, not about ClamAV release cycles. Besides, you miss another point: ClamAV is an open source software, consisting of roughly 150K lines of C code and 65 signatures, currently maintained by three full time developers, one and a half full time sigmakers and a system administrator. We ALWAYS ask our users to test the development head and provide feedbacks because we cannot do it all on our own: we lack the man power and we lack the infrastructure, but, most importantly we lack YOUR setup, YOUR deployment and YOUR envirnonment. With some very notable exceptions (which I would really like to thank), it is a fact that, despite the repeated requests, not many people test the code. You can look at the bugzilla being all quiet for weeks, then, as soon as we release a new version, it suddently gets flooded with tickets. So, to conclude, if you want to get better releases, do your bit. The only alternative is that we release what WE think is ok and we re-release when YOU tell us it's not. Thanks for the lesson, -aCaB ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
Hi there, On Thu, 29 Oct 2009 aCaB wrote: On our side we do a lot of QA ... I really believe something needs to happen here so that these type of bugs can be caught quickly before they affect a number of users. Thoughts? I suspect that rather than QA, what you do is just a lot of hap-hazard testing. That's why, whenever I see a new release of ClamAV, first I will suppress a groan and then, before I risk it on any of my servers, I'll wait a while and watch the users' list to see how much trouble it causes. This approach serves me well, although I can't say I'm proud of the fact that I'm letting a lot of poor innocents do my acceptance testing for me. To be brutally frank, if I upgraded ClamAV on the basis of the release announcements, ClamAV would in my installations cause a lot more trouble than the things which it's designed to combat. You need to develop a proper QA system. Briefly, that means you need to document exactly how you're going to avoid this sort of chaos, and then you have to do what it says in the documentation that you're going to do. There's a fairly famous transcript of some evidence to a House of Commons committee around here somewhere... ah, here it is: [excerpt] I would like to tell you something that you will not believe but which I think it is important that you hear, and that is that almost every IT supplier in the world today is incompetent. I have worked in the IT industry almost all my working life for large and small organisations, and I know what I speak. For example, the typical rate of delivered faults after full user acceptance testing from the maker suppliers in the industry over many years has been steady at around 20 faults per thousand lines of code. We know how to deliver software with a fault rate that is down around 0.1 faults per thousand lines of code and the industry does not adopt these techniques. We are as an industry very much in the early stages. The industry is only 50 years old. If you compare that with civil engineering, which is several thousand years old, we are tackling some of the most complex engineering designs and building some of the most complex engineering systems that the world has ever seen, essentially using craft technology. If you looked at the methods that are employed in most companies you would come to the conclusion that actually IT system development is a fashion business, not an engineering business, because they jump from one methodology to another year after year so long as it has a whizzy name, Agile this or Intensive that. The underlying engineering disciplines that every mature engineering discipline has learnt it needs to use in order to be able to show that the system it is building has the required properties have not yet been employed in software and systems engineering, and that is at the heart of why these things do not work. [/excerpt] Professor Martyn Thomas, Minutes of Evidence taken before Home Affairs Committee Inquiry into Identity Cards, Tuesday 24 February 2004. http://www.parliament.the-stationery-office.co.uk/pa/cm200304/cmselect/cmhaff/uc130-iv/uc13002.htm -- 73, Ged. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
Steve Basford wrote: LibClamAV Error: mpool_malloc(): Attempt to allocate 2097152 bytes. Please report to http://bugs.clamav.net LibClamAV Error: cli_ac_addpatt: Can't realloc ac_pattable LibClamAV Error: cli_parse_add(): Thanks to the ClamAV team, the bug was fixed in the clamav-devel version: clamav-devel: +Sat Oct 24 15:06:50 CEST 2009 (acab) + * libclamav/mpool.c: increase max pool to 8M to allow loading huge custom dbs Hi Steve, The (now) increased pool size is around 16 times bigger than the largest pool used by the offical db, so it'll probably be ok for a while. That said, we should still figure out a way to avoid this kind of troubles in the future (same goes for the infamous clamd crashes while loading 3rd party db's bug which plagued the early 0.95's). On our side we do a lot of QA over our own signatures to make sure things like that won't happen, but of course we can't guarantee the same for 3rd party databases. At the end of the day, any service disruption, even if caused by the use custom databases, is problematic and affects the entire ClamAV user community. I'm wondering if it would make sense for us to open up the QA side of our infrastructure to you guys, in order to minimize this kind of inconvenence. I really believe something needs to happen here so that these type of bugs can be caught quickly before they affect a number of users. Thoughts? aCaB ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
On Sun, Oct 25, 2009 at 11:27 AM, Steve Basford steveb_cla...@sanesecurity.com wrote: Hi All, Some users (mainly x86_64 so far) noticed database errors (malformed database) when loading signatures. As signature integrity is checked before upload to the mirrors and the download scripts check integrity before use, this issue should not arise. With help from various people on the Sanesecurity list, the problem was narrowed down to users on x86_64 os versions eg: CentOS 5.4 on x86_64, who were using nearly all the available Third Party databases. The typical errors were: LibClamAV Error: mpool_malloc(): Attempt to allocate 2097152 bytes. Please report to http://bugs.clamav.net LibClamAV Error: cli_ac_addpatt: Can't realloc ac_pattable LibClamAV Error: cli_parse_add(): Thanks to the ClamAV team, the bug was fixed in the clamav-devel version: clamav-devel: +Sat Oct 24 15:06:50 CEST 2009 (acab) + * libclamav/mpool.c: increase max pool to 8M to allow loading huge custom dbs I realise that people may not be able to move to the devel version in production environments, so the only work-around is to try and limit the number of databases that you are using for example: Largest size signature databases: 25/10/2009 15:53 2,526,656 jurlbl.ndb 24/10/2009 16:53 3,082,316 junk.ndb 25/10/2009 15:38 3,327,576 INetMsg-SpamDomains-2w.ndb 25/10/2009 15:29 3,886,074 scamnailer.ndb 25/10/2009 15:53 6,967,926 jurlbla.ndb 28/08/2009 12:10 9,393,566 securiteinfo.hdb 25/10/2009 15:47 12,645,831 INetMsg-SpamDomains-2m.ndb As a reminder if you are using InetMsg signatures, you need to select: *either* INetMsg-SpamDomains-2w.ndb *or* INetMsg-SpamDomains-2m.ndb *not* both to save a bit of memory. Hopefully once the devel version bugfix makes it's way into the stable version, this problem should go away. Sorry for any problems this has caused. Cheers, Steve Sanesecurity ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml We are using Ubuntu 9.04 x86_64. What symptoms were observed when they noticed database errors? -- Robert Lopez Unix Systems Administrator Central New Mexico Community College (CNM) 525 Buena Vista SE Albuquerque, New Mexico 87106 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
[Clamav-users] [Fwd: [sanesecurity] x86_64 users: possible malformed database problems]
Hi All, Some users (mainly x86_64 so far) noticed database errors (malformed database) when loading signatures. As signature integrity is checked before upload to the mirrors and the download scripts check integrity before use, this issue should not arise. With help from various people on the Sanesecurity list, the problem was narrowed down to users on x86_64 os versions eg: CentOS 5.4 on x86_64, who were using nearly all the available Third Party databases. The typical errors were: LibClamAV Error: mpool_malloc(): Attempt to allocate 2097152 bytes. Please report to http://bugs.clamav.net LibClamAV Error: cli_ac_addpatt: Can't realloc ac_pattable LibClamAV Error: cli_parse_add(): Thanks to the ClamAV team, the bug was fixed in the clamav-devel version: clamav-devel: +Sat Oct 24 15:06:50 CEST 2009 (acab) + * libclamav/mpool.c: increase max pool to 8M to allow loading huge custom dbs I realise that people may not be able to move to the devel version in production environments, so the only work-around is to try and limit the number of databases that you are using for example: Largest size signature databases: 25/10/2009 15:53 2,526,656 jurlbl.ndb 24/10/2009 16:53 3,082,316 junk.ndb 25/10/2009 15:38 3,327,576 INetMsg-SpamDomains-2w.ndb 25/10/2009 15:29 3,886,074 scamnailer.ndb 25/10/2009 15:53 6,967,926 jurlbla.ndb 28/08/2009 12:10 9,393,566 securiteinfo.hdb 25/10/2009 15:4712,645,831 INetMsg-SpamDomains-2m.ndb As a reminder if you are using InetMsg signatures, you need to select: *either* INetMsg-SpamDomains-2w.ndb *or* INetMsg-SpamDomains-2m.ndb *not* both to save a bit of memory. Hopefully once the devel version bugfix makes it's way into the stable version, this problem should go away. Sorry for any problems this has caused. Cheers, Steve Sanesecurity ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
