Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 31/10/2019 12:04, Reio Remma wrote: On 28/10/2019 12:55, Reio Remma via clamav-users wrote: On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote: Hi Micah, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote: I'm sorry, Ged... Apology accepted. :) I'm now running the development (0.102) version of clamd, patched with Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched one, and one with the patch that I posted on Bugzilla). The milter scans all mail with all three daemons. On the arrival of a message, if the database is not already being reloaded I start a fresh reload before the scan so that, for all scans, a reload always executes concurrently. Nothing seems to have broken, and so far there's nothing terribly interesting to report other than the strange failure to detect which I sent to Joel early this week (and which I'm sure has nothing to do with these patches). I've been running a patched 101.4 for a few weeks now and unfortunately I'm observing a memory leak from the multithreaded database reloads. I'm observing clamd memory usage going up when the new database loads and then eventually dropping down to 1.3G again. For some reason "eventually" means the memory usage drops down only after clamd processes the next e-mail. The problem however shows itself if clamd happens to reload its database 2 times if a row with no mail processed in between. Seemingly it will have 3 databases in memory then and the next mail being processed releases one of them, but the extra database will remain "somewhere". All sorts of weird problems always keep popping up on due to low traffic on the server. :) Fortunately 0.102.0 with the patch from ClamAV team doesn't have that issue and seems to release the extra memory right away. Happily running 0.102.0 now. Has anyone got the threaded reload patch working with 0.102.2? When rebuilding my RPM with 0.102.2, I get the following error when the patch is being applied: + echo 'Patch #0 (clamd-threaded-reloading.patch):' Patch #0 (clamd-threaded-reloading.patch): + /usr/bin/cat ~/rpmbuild/SOURCES/clamd-threaded-reloading.patch + /usr/bin/patch -p1 -b --suffix .threaded_reloading --fuzz=0 patching file clamd/clamd.c Reversed (or previously applied) patch detected! Assume -R? [n] Thanks, Reio ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 28/10/2019 12:55, Reio Remma via clamav-users wrote: On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote: Hi Micah, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote: I'm sorry, Ged... Apology accepted. :) I'm now running the development (0.102) version of clamd, patched with Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched one, and one with the patch that I posted on Bugzilla). The milter scans all mail with all three daemons. On the arrival of a message, if the database is not already being reloaded I start a fresh reload before the scan so that, for all scans, a reload always executes concurrently. Nothing seems to have broken, and so far there's nothing terribly interesting to report other than the strange failure to detect which I sent to Joel early this week (and which I'm sure has nothing to do with these patches). I've been running a patched 101.4 for a few weeks now and unfortunately I'm observing a memory leak from the multithreaded database reloads. I'm observing clamd memory usage going up when the new database loads and then eventually dropping down to 1.3G again. For some reason "eventually" means the memory usage drops down only after clamd processes the next e-mail. The problem however shows itself if clamd happens to reload its database 2 times if a row with no mail processed in between. Seemingly it will have 3 databases in memory then and the next mail being processed releases one of them, but the extra database will remain "somewhere". All sorts of weird problems always keep popping up on due to low traffic on the server. :) Fortunately 0.102.0 with the patch from ClamAV team doesn't have that issue and seems to release the extra memory right away. Happily running 0.102.0 now. Good luck, Reio ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi Reio, On Mon, 28 Oct 2019, Reio Remma via clamav-users wrote: ... I've been running a patched 101.4 for a few weeks now and unfortunately I'm observing a memory leak from the multithreaded database reloads. I'm observing clamd memory usage going up when the new database loads ... The problem however shows itself if clamd happens to reload its database 2 times if a row with no mail processed in between. Seemingly it will have 3 databases in memory then and the next mail being processed releases one of them, but the extra database will remain "somewhere". .. As I said I'm using 0.102-rc with the older patch, and I haven't seen this behaviour (but I have been looking for it, and anything like it, using Nagios etc.). On our servers there's no risk of clamd reloading databases without processing a message inbetween the reloads, but I'm sure I could arrange it if neccessary. :) Unfortunately at the moment I have no time to investigate but I guess it will be simple to fix if it isn't something peculiar to your setup - for example it might be a problem with threads in a library. From my reading of the code, going back admittedly a little while now, it seemed very clear that the old database should be freed unconditionally after the new one was loaded. I'd suggest that you raise an issue in the ClamAV Bugzilla. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote: Hi Micah, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote: I'm sorry, Ged... Apology accepted. :) I'm now running the development (0.102) version of clamd, patched with Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched one, and one with the patch that I posted on Bugzilla). The milter scans all mail with all three daemons. On the arrival of a message, if the database is not already being reloaded I start a fresh reload before the scan so that, for all scans, a reload always executes concurrently. Nothing seems to have broken, and so far there's nothing terribly interesting to report other than the strange failure to detect which I sent to Joel early this week (and which I'm sure has nothing to do with these patches). I've been running a patched 101.4 for a few weeks now and unfortunately I'm observing a memory leak from the multithreaded database reloads. I'm observing clamd memory usage going up when the new database loads and then eventually dropping down to 1.3G again. For some reason "eventually" means the memory usage drops down only after clamd processes the next e-mail. The problem however shows itself if clamd happens to reload its database 2 times if a row with no mail processed in between. Seemingly it will have 3 databases in memory then and the next mail being processed releases one of them, but the extra database will remain "somewhere". All sorts of weird problems always keep popping up on due to low traffic on the server. :) Reio ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
> One thing we could do is have clamd "start" before loading the database. > That is to say that it would immediately begin listening on the unix/tcp > socket > for requests and fork into the background so as not to block the boot process. > All scan requests would then be blocked while the database loads. > I imagine this would solve most of the frustration around boot-up load time I guess I kind of jumped the gun on this one, chalk it up to the late-night message posting... While it is an older linux distro with the init startup, I simply moved ClamAV to near the end of the boot process... problem solved. I noticed some people made that recommendation too. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi Micah, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote: I'm sorry, Ged... Apology accepted. :) I'm now running the development (0.102) version of clamd, patched with Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched one, and one with the patch that I posted on Bugzilla). The milter scans all mail with all three daemons. On the arrival of a message, if the database is not already being reloaded I start a fresh reload before the scan so that, for all scans, a reload always executes concurrently. Nothing seems to have broken, and so far there's nothing terribly interesting to report other than the strange failure to detect which I sent to Joel early this week (and which I'm sure has nothing to do with these patches). -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
* Micah Snyder via clamav-users: > [ClamAV] would immediately begin listening on the unix/tcp socket for > requests and fork into the background so as not to block the boot > process. To me, slowing down the boot process is just the (admittedly annoying) symptom of an underlying ClamAV issue. Based on the delays that we have seen over the past months, I'd say that ClamAV's database handling does not scale well enough, and I think that's what needs fixing. -Ralph ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
I'm sorry, Ged. I didn't mean to demean the work of Julius Plenz, asulfrian, or yourself. I stepped into my current position on the ClamAV team just over two years ago and in my time here there have been many tasks that have been on the backburner or ignored entirely for multiple years. We have been working to try to rectify this but it is a slow process. I do appreciate your work to try to update and test the original patch. Alberto saw the discussions here demonstrating a need for this feature. He offered to share a similar patch he had working in production on his systems, one that was updated to apply cleanly with the 0.102 code base. I wanted to share it immediately with you all. You're right that I should've given more credit to the authors of the prior work. It was only my intention to credit Alberto so as not to give the false impression that his work was my own. Regards, Micah On 9/13/19, 11:15 AM, "clamav-users on behalf of G.W. Haywood via clamav-users" wrote: Hi there, On Thu, 12 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote: > https://bugzilla.clamav.net/show_bug.cgi?id=10979#c19 > This patch applies to the current head of dev/0.102 ... If the development version is a step too far, the two files which I posted on September 10th implement a patch which has been sitting on the ClamAV Bugzilla (at #c2) for nearly three years: https://bugzilla.clamav.net/show_bug.cgi?id=10979#c13 https://bugzilla.clamav.net/show_bug.cgi?id=10979#c14 These replace two files in the current (v0.101.4) release, to produce results very similar to those from the patch at #c19 for v0.102.x. Unfortunately there are so many cosmetic changes in the development version that a direct comparison of the patches might be tedious, but the essentials are the same. Load new data in a separate thread, and in the meantime scan using the old database; switch database pointers (virtually instantaneous) on reload completion; ignore database reload requests if reloading is already in progress; and when the old data is no longer needed, drop it. Test results and/or observations welcome. This will not of course help start-up times at all, but it's easy to arrange to load a smaller database at startup if that's what you feel you must do - there has been a discussion about using what I'll call non-standard databases recently. Personally I don't see the need for anything like that; the runtimes of my clamd daemons are rarely less than months, even if I'm testing things, so it's of no consequence if loading the data at the beginning of a run takes a couple of minutes. Since I'm only scanning mail, rather than scan it with less than the full deck I'll just delay it a couple of minutes. Until I worked on this patch, that's what I'd been doing on every database reload and, as I've always maintained, it's really no big deal. > ...do not confuse the fact that we are paid with the thought that > you are paying us. I'm not sure that ham-fisted attempt at a justification was entirely called for, Micah. You had a patch for several years. Then, two and a half days after I posted the two files shown above, you're galvanized into action; but you studiously avoid mention of the prior work by several people, and then imply that people are confused when everything is crystal clear. > We of course always appreciate help from the community ... Perhaps you could try to make it a little more obvious. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Ged, That's a fair assessment. This is why I asked. Thanks, Micah On 9/13/19, 11:26 AM, "clamav-users on behalf of G.W. Haywood via clamav-users" wrote: Hi there, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote: > One thing we could do is have clamd "start" before loading the > database. That is to say that it would immediately begin listening > on the unix/tcp socket for requests and fork into the background so > as not to block the boot process. All scan requests would then be > blocked while the database loads. I imagine this would solve most > of the frustration around boot-up load time. I don't think you should be trying to second-guess stuff like this, and I don't quite see how in these days of parallel boot processes that anything will get blocked that doesn't need to be blocked. Will you be looking at the network interfaces? The routes? You'll end up writing another systemd. The system administrator/integrator needs to earn his living somehow; not asking a utility to do things when it's not yet ready to do them is one of his jobs. It's why there are all those symlinks in /etc/rc3.d/. > Does this have any appeal? Seems like a waste of effort to me. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote: One thing we could do is have clamd "start" before loading the database. That is to say that it would immediately begin listening on the unix/tcp socket for requests and fork into the background so as not to block the boot process. All scan requests would then be blocked while the database loads. I imagine this would solve most of the frustration around boot-up load time. I don't think you should be trying to second-guess stuff like this, and I don't quite see how in these days of parallel boot processes that anything will get blocked that doesn't need to be blocked. Will you be looking at the network interfaces? The routes? You'll end up writing another systemd. The system administrator/integrator needs to earn his living somehow; not asking a utility to do things when it's not yet ready to do them is one of his jobs. It's why there are all those symlinks in /etc/rc3.d/. Does this have any appeal? Seems like a waste of effort to me. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 13.09.19 14:42, Micah Snyder (micasnyd) via clamav-users wrote: One thing we could do is have clamd "start" before loading the database. That is to say that it would immediately begin listening on the unix/tcp socket for requests and fork into the background so as not to block the boot process. All scan requests would then be blocked while the database loads. I imagine this would solve most of the frustration around boot-up load time. Does this have any appeal? on debian we've had parallel startup for some time, systems using systemd should have that one too (but maybe it runs on foreground there). I'd personally expect clamd to be fully working after it forks at startup. other SW may expect that too. Maybe with other startup parameter? On 9/12/19, 11:31 PM, "clamav-users on behalf of J.R. via clamav-users" wrote: This patch will be a very welcome addition! Oddly enough today my hosting company had an emergency and I needed to shutdown my server so it could be physically moved mid-day. The painfully slow load time of ClamAV was excruciating apparent while I was watching the console slowly go through the boot process. While a second thread to *reload* the database in the background is going to be a nice feature, I would assume it wouldn't help any on initial startup. While tweaking things with this 2nd thread, maybe there could be a start-up option / flag to only load like the daily.cld (or official sigs only) to minimize blocking on boot-up, but still allow a decent level of protection. Then a full DB could be loading up in its separate thread and swapped when ready? I honestly have no idea how the signatures load, but would a full multi-threaded model even theoretically work? Or would that not allow correct parsing / loading of the signatures? It just seems with PCs and servers having so many cores, and the number of viruses ever-increasing... Alternatively, would there be a way to do a "diff" on the loaded signatures in memory to add / remove only the ones that have changed (when feasible over a full reload)? Seems like an awful lot of unnecessary re-parsing is being done when only a small handful of signatures are added at any given time. Just throwing some ideas out there... Always thankful for all the hard work from the development team. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. I just got lost in thought. It was unfamiliar territory. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Thu, 12 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote: https://bugzilla.clamav.net/show_bug.cgi?id=10979#c19 This patch applies to the current head of dev/0.102 ... If the development version is a step too far, the two files which I posted on September 10th implement a patch which has been sitting on the ClamAV Bugzilla (at #c2) for nearly three years: https://bugzilla.clamav.net/show_bug.cgi?id=10979#c13 https://bugzilla.clamav.net/show_bug.cgi?id=10979#c14 These replace two files in the current (v0.101.4) release, to produce results very similar to those from the patch at #c19 for v0.102.x. Unfortunately there are so many cosmetic changes in the development version that a direct comparison of the patches might be tedious, but the essentials are the same. Load new data in a separate thread, and in the meantime scan using the old database; switch database pointers (virtually instantaneous) on reload completion; ignore database reload requests if reloading is already in progress; and when the old data is no longer needed, drop it. Test results and/or observations welcome. This will not of course help start-up times at all, but it's easy to arrange to load a smaller database at startup if that's what you feel you must do - there has been a discussion about using what I'll call non-standard databases recently. Personally I don't see the need for anything like that; the runtimes of my clamd daemons are rarely less than months, even if I'm testing things, so it's of no consequence if loading the data at the beginning of a run takes a couple of minutes. Since I'm only scanning mail, rather than scan it with less than the full deck I'll just delay it a couple of minutes. Until I worked on this patch, that's what I'd been doing on every database reload and, as I've always maintained, it's really no big deal. ...do not confuse the fact that we are paid with the thought that you are paying us. I'm not sure that ham-fisted attempt at a justification was entirely called for, Micah. You had a patch for several years. Then, two and a half days after I posted the two files shown above, you're galvanized into action; but you studiously avoid mention of the prior work by several people, and then imply that people are confused when everything is crystal clear. We of course always appreciate help from the community ... Perhaps you could try to make it a little more obvious. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
One thing we could do is have clamd "start" before loading the database. That is to say that it would immediately begin listening on the unix/tcp socket for requests and fork into the background so as not to block the boot process. All scan requests would then be blocked while the database loads. I imagine this would solve most of the frustration around boot-up load time. Does this have any appeal? -Micah On 9/12/19, 11:31 PM, "clamav-users on behalf of J.R. via clamav-users" wrote: This patch will be a very welcome addition! Oddly enough today my hosting company had an emergency and I needed to shutdown my server so it could be physically moved mid-day. The painfully slow load time of ClamAV was excruciating apparent while I was watching the console slowly go through the boot process. While a second thread to *reload* the database in the background is going to be a nice feature, I would assume it wouldn't help any on initial startup. While tweaking things with this 2nd thread, maybe there could be a start-up option / flag to only load like the daily.cld (or official sigs only) to minimize blocking on boot-up, but still allow a decent level of protection. Then a full DB could be loading up in its separate thread and swapped when ready? I honestly have no idea how the signatures load, but would a full multi-threaded model even theoretically work? Or would that not allow correct parsing / loading of the signatures? It just seems with PCs and servers having so many cores, and the number of viruses ever-increasing... Alternatively, would there be a way to do a "diff" on the loaded signatures in memory to add / remove only the ones that have changed (when feasible over a full reload)? Seems like an awful lot of unnecessary re-parsing is being done when only a small handful of signatures are added at any given time. Just throwing some ideas out there... Always thankful for all the hard work from the development team. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-09-13 00:57, schrieb Micah Snyder (micasnyd) via clamav-users: Henrik, all: Alberto Wu, a former ClamAV core developer, has very kindly provided a patch to us today that he created to perform clamd database reloads in a separate thread. I have attached his work, with some minor tweaks on my part, to the relevant Bugzilla ticket, here: Big thanks to Alberto Wu and you! I ll be happy too about this new feature . This issue motivated me to get into C-Programming again in my spare-time :) I ve some problems to concentrate on pointer to pointer to "tries structure", but it s getting better. I have fun trying to understand the code work. Regards Thomas Barth ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
This patch will be a very welcome addition! Oddly enough today my hosting company had an emergency and I needed to shutdown my server so it could be physically moved mid-day. The painfully slow load time of ClamAV was excruciating apparent while I was watching the console slowly go through the boot process. While a second thread to *reload* the database in the background is going to be a nice feature, I would assume it wouldn't help any on initial startup. While tweaking things with this 2nd thread, maybe there could be a start-up option / flag to only load like the daily.cld (or official sigs only) to minimize blocking on boot-up, but still allow a decent level of protection. Then a full DB could be loading up in its separate thread and swapped when ready? I honestly have no idea how the signatures load, but would a full multi-threaded model even theoretically work? Or would that not allow correct parsing / loading of the signatures? It just seems with PCs and servers having so many cores, and the number of viruses ever-increasing... Alternatively, would there be a way to do a "diff" on the loaded signatures in memory to add / remove only the ones that have changed (when feasible over a full reload)? Seems like an awful lot of unnecessary re-parsing is being done when only a small handful of signatures are added at any given time. Just throwing some ideas out there... Always thankful for all the hard work from the development team. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Henrik, all: Alberto Wu, a former ClamAV core developer, has very kindly provided a patch to us today that he created to perform clamd database reloads in a separate thread. I have attached his work, with some minor tweaks on my part, to the relevant Bugzilla ticket, here: https://bugzilla.clamav.net/show_bug.cgi?id=10979#c19 This patch applies to the current head of dev/0.102 of our Github repository: https://github.com/Cisco-Talos/clamav-devel Thank so much Alberto for submitting the patch. We would appreciate feedback from any who are willing and able to test the new feature. Regarding the conversation below... We don't mind nudging or even a little nagging. In fact, hearing from a wide number of community members on specific issues is the best way for us to know which bugs or features most need our attention. That said, do not confuse the fact that we are paid with the thought that you are paying us. We have a variety of customers both internal and external and we must prioritize our limited resources to both fix known issues and add new features for all of our users. We of course always appreciate help from the community, such as the clamd database reloading feature patch above. Respectfully, Micah Micah Snyder ClamAV Development Talos Cisco Systems, Inc. On 8/31/19, 12:54 PM, "clamav-users on behalf of Scott Kitterman via clamav-users" wrote: On August 31, 2019 4:32:00 PM UTC, Henrik K wrote: >On Sat, Aug 31, 2019 at 12:21:11PM -0400, Scott Kitterman via >clamav-users wrote: >> >> Not to put too fine a point on it, but if you are unhappy with the >service you >> are receiving, you should switch to a different vendor. I suspect >it's >> unlikely you'll get the same value for money elsewhere. > >Does this worn cliche really need posting? :-) > >But hey, I'm just participating in the community.. sometimes things >just >need a bit of nudging. I wouldn't even continue to nag about it, if >this >was a basic volunteer project. But we are talking about a security >company >that should be proud of it's code. There's no problem with nudging, but being nasty about isn't appropriate. I'll confess that I aimed a comment at you that should have been pointed at the OP. Sorry about that. I've been maintaining clamav packages for over a decade through three different companies owning the project and overall I think the Talos/Cisco people are doing a pretty good job. They are generally responsive. Also, in terms of being proud of their code, you should compare the code quality when Sourcefire bought clamav to the current code base. It's night and day different. Also the rate of security bugs seems to have dropped off (not to mention we actually get bugfix releases now). So yeah, they could do better, but looking back, I think they're doing pretty good. In this case, the 'worn cliche' is important, because unlike lots of other FOSS projects, this one doesn't have non-proprietary alternatives, so it'd be pretty awful if the community were to convince Talos that publishing it was more trouble than it was worth. Scott K ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-09-07 21:03, schrieb Robert M. Stockmann via clamav-users: Why everyone needs two minutes for this task, independent from which hardware is used, is a puzzle to me. Anyone who has the clamd .cvd files loaded on a fast SSD storage ? I also use unofficial signatures from several vendors. Today someone decided to clean out signatures. It s a bit faster now. Sat Sep 7 01:13:27 2019 -> Reading databases from /var/lib/clamav Sat Sep 7 01:15:21 2019 -> Database correctly reloaded (10999143 signatures) -- Sat Sep 7 03:05:33 2019 -> Reading databases from /var/lib/clamav Sat Sep 7 03:07:29 2019 -> Database correctly reloaded (10998591 signatures) -- Sat Sep 7 05:08:55 2019 -> Reading databases from /var/lib/clamav Sat Sep 7 05:10:41 2019 -> Database correctly reloaded (10869402 signatures) -- Sat Sep 7 07:10:31 2019 -> Reading databases from /var/lib/clamav Sat Sep 7 07:12:18 2019 -> Database correctly reloaded (10869489 signatures) I use virtual Servers with Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 16 GB RAM and 100% SSD. I could download every hour because I get a 0 hour signature db. But I've set the interval for the checks much lower during office hours and higher during out of office hours to minimize the chance of a collision between reloading and sending an e-Mail. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Sat, Sep 07, 2019 at 13:06 PM, Robert M. Stockmann via clamav-users wrote: > On Sat, 7 Sep 2019, Reio Remma via clamav-users wrote: >> Date: Sat, 7 Sep 2019 22:57:08 +0300 >> From: Reio Remma via clamav-users > <mailto:clamav-users@lists.clamav.net>> >> To: clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net> >> Cc: Reio Remma mailto:r...@mrstuudio.ee>> >> Subject: Re: [clamav-users] How to boost clamav? Reloading database >>results in a talking timeout? >> >> On 07.09.2019 22:03, Robert M. Stockmann via clamav-users wrote: >>> On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: >>>>>>> I guess many of us are just running too old hardware. :) >>>>>>> >>>>>>> Here's a comparison between my mail server and identical config >>>>>>> running in a VM. >>>>>>> >>>>>>> Sep 6 09:41:06 mail clamd[31441]: Reading databases from >>>>>>> /var/lib/clamav >>>>>>> Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded >>>>>>> (10741767 ... >>>>>>> >>>>>>> Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav >>>>>>> Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 >>>>>>> ... >>>> >>> Why everyone needs two minutes for this task, independent from which >>> hardware is used, is a puzzle to me. Anyone who has the clamd .cvd >>> files loaded on a fast SSD storage ? >> >> My original point was that its heavily CPU bound. >> >> As you can see from the logs I initally posted the speed difference >> between a CPU from 2005 and 2019 is 6 times (3 minutes vs 30 seconds). >> > > So what about the number of correctly reloaded signatures ? I notice > you have about 10 million, someone else reported 8 million signatures > as of Sept 6, I currently read from my clamd logfile : > > Sat Sep 7 10:54:10 2019 -> SelfCheck: Database status OK. > Sat Sep 7 11:04:13 2019 -> SelfCheck: Database status OK. > Sat Sep 7 11:08:59 2019 -> Reading databases from /var/lib/clamav > Sat Sep 7 11:11:07 2019 -> Database correctly reloaded (6309062 signatures) > Sat Sep 7 11:21:07 2019 -> SelfCheck: Database status OK. > Sat Sep 7 11:31:08 2019 -> SelfCheck: Database status OK. The number will vary depending on whether or not you include unofficial signatures and the settings in freshclam.conf regarding what types of files you choose to scan or skip. -Al- smime.p7s Description: S/MIME cryptographic signature ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Sat, 7 Sep 2019, Reio Remma via clamav-users wrote: > Date: Sat, 7 Sep 2019 22:57:08 +0300 > From: Reio Remma via clamav-users > To: clamav-users@lists.clamav.net > Cc: Reio Remma > Subject: Re: [clamav-users] How to boost clamav? Reloading database > results in a talking timeout? > > On 07.09.2019 22:03, Robert M. Stockmann via clamav-users wrote: > > On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: > >>>>> I guess many of us are just running too old hardware. :) > >>>>> > >>>>> Here's a comparison between my mail server and identical config > >>>>> running in a VM. > >>>>> > >>>>> Sep 6 09:41:06 mail clamd[31441]: Reading databases from > >>>>> /var/lib/clamav > >>>>> Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded > >>>>> (10741767 ... > >>>>> > >>>>> Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav > >>>>> Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 > >>>>> ... > >> > > Why everyone needs two minutes for this task, independent from which > > hardware is used, is a puzzle to me. Anyone who has the clamd .cvd > > files loaded on a fast SSD storage ? > > My original point was that its heavily CPU bound. > > As you can see from the logs I initally posted the speed difference > between a CPU from 2005 and 2019 is 6 times (3 minutes vs 30 seconds). > So what about the number of correctly reloaded signatures ? I notice you have about 10 million, someone else reported 8 million signatures as of Sept 6, I currently read from my clamd logfile : Sat Sep 7 10:54:10 2019 -> SelfCheck: Database status OK. Sat Sep 7 11:04:13 2019 -> SelfCheck: Database status OK. Sat Sep 7 11:08:59 2019 -> Reading databases from /var/lib/clamav Sat Sep 7 11:11:07 2019 -> Database correctly reloaded (6309062 signatures) Sat Sep 7 11:21:07 2019 -> SelfCheck: Database status OK. Sat Sep 7 11:31:08 2019 -> SelfCheck: Database status OK. -- Robert M. Stockmann - RHCE Network Engineer - UNIX/Linux Specialist crashrecovery.org st...@stokkie.net ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 07.09.2019 22:03, Robert M. Stockmann via clamav-users wrote: On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep 6 09:41:06 mail clamd[31441]: Reading databases from /var/lib/clamav Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded (10741767 ... Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 ... Why everyone needs two minutes for this task, independent from which hardware is used, is a puzzle to me. Anyone who has the clamd .cvd files loaded on a fast SSD storage ? My original point was that its heavily CPU bound. As you can see from the logs I initally posted the speed difference between a CPU from 2005 and 2019 is 6 times (3 minutes vs 30 seconds). Good luck, Reio ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
>>On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: >>>I guess many of us are just running too old hardware. :) >>> >>>Here's a comparison between my mail server and identical config >>>running in a VM. >>> >>>Sep 6 09:41:06 mail clamd[31441]: Reading databases from >>>/var/lib/clamav >>>Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded >>>(10741767 ... >>> >>>Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav >>>Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 ... On Fri, 6 Sep 2019, Matus UHLAR - fantomas wrote: Fri Sep 6 08:49:08 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 08:50:18 2019 -> Database correctly reloaded (8830356 signatures) Fri Sep 6 09:48:25 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 09:49:49 2019 -> Database correctly reloaded (8830677 signatures) Fri Sep 6 10:47:36 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 10:48:53 2019 -> Database correctly reloaded (8830954 signatures) average ~1:20 on X3440 CPU (10 years old). On 07.09.19 21:03, Robert M. Stockmann via clamav-users wrote: I notice that the above clamd configuration is reloading the clamav databases every hour, but thats not how its supposed to work. My configuration only attempts a reload when actually new .cvd updates are received : If you read the logs carefully, you'd notice that there's different number of signatures, thus, the database did change. That's not the poing. >On 06/09/2019 11:31, G.W. Haywood wrote: >>That's very useful, thanks. Can you compare the costs of running >>them for us? On 06.09.19 11:54, Reio Remma via clamav-users wrote: >I suspect the i9-9900 is cheaper to actually run than the old >whichever Core is in the mail server. :D I think that virtual/cloud server has to be cheaper than power usage of the existing server (plus housing, if you pay for that one). Why everyone needs two minutes for this task, independent from which hardware is used, is a puzzle to me. Anyone who has the clamd .cvd files loaded on a fast SSD storage ? I think it needs fast CPU, that's why it takes time. all the signatures have to be parsed and indexed in memory and that's huge number of signatures. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. WinError #98652: Operation completed successfully. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Fri, 6 Sep 2019, Matus UHLAR - fantomas wrote: > Date: Fri, 6 Sep 2019 11:00:20 +0200 > From: Matus UHLAR - fantomas > Reply-To: ClamAV users ML > To: clamav-users@lists.clamav.net > Subject: Re: [clamav-users] How to boost clamav? Reloading database > results in a talking timeout? > > >>On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: > >>>I guess many of us are just running too old hardware. :) > >>> > >>>Here's a comparison between my mail server and identical config > >>>running in a VM. > >>> > >>>Sep 6 09:41:06 mail clamd[31441]: Reading databases from > >>>/var/lib/clamav > >>>Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded > >>>(10741767 ... > >>> > >>>Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav > >>>Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 ... > > Fri Sep 6 08:49:08 2019 -> Reading databases from /var/lib/clamav > Fri Sep 6 08:50:18 2019 -> Database correctly reloaded (8830356 signatures) > Fri Sep 6 09:48:25 2019 -> Reading databases from /var/lib/clamav > Fri Sep 6 09:49:49 2019 -> Database correctly reloaded (8830677 signatures) > Fri Sep 6 10:47:36 2019 -> Reading databases from /var/lib/clamav > Fri Sep 6 10:48:53 2019 -> Database correctly reloaded (8830954 signatures) > > average ~1:20 on X3440 CPU (10 years old). I notice that the above clamd configuration is reloading the clamav databases every hour, but thats not how its supposed to work. My configuration only attempts a reload when actually new .cvd updates are received : from freshclam.log : -- ClamAV update process started at Sun Sep 1 10:07:00 2019 main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) daily.cld is up to date (version: 25558, sigs: 1744125, f-level: 63, builder: raynman) bytecode.cvd is up to date (version: 330, sigs: 94, f-level: 63, builder: neo) -- ClamAV update process started at Sun Sep 1 11:07:00 2019 main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) Downloading daily-25559.cdiff [100%] daily.cld updated (version: 25559, sigs: 1745720, f-level: 63, builder: raynman) Can't query daily.25559.105.1.0.6810DA54.ping.clamav.net bytecode.cvd is up to date (version: 330, sigs: 94, f-level: 63, builder: neo) Database updated (6312063 signatures) from db.nl.clamav.net (IP: 104.16.218.84) Clamd successfully notified about the update. -- ClamAV update process started at Sun Sep 1 12:07:00 2019 main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) daily.cld is up to date (version: 25559, sigs: 1745720, f-level: 63, builder: raynman) bytecode.cvd is up to date (version: 330, sigs: 94, f-level: 63, builder: neo) -- At 11:07 the ClamAV update process (run hourly with 7 * * * * /usr/bin/freshclam --quiet) is started which actually has new updates : daily-25559.cdiff. Next freshclam notifies clamd : "Clamd successfully notified about the update." clamd.log from the same time period shows this : Sun Sep 1 09:56:55 2019 -> SelfCheck: Database status OK. Sun Sep 1 10:15:01 2019 -> SelfCheck: Database status OK. Sun Sep 1 10:33:03 2019 -> SelfCheck: Database status OK. Sun Sep 1 10:43:04 2019 -> SelfCheck: Database status OK. Sun Sep 1 10:58:44 2019 -> SelfCheck: Database status OK. Sun Sep 1 11:08:54 2019 -> SelfCheck: Database modification detected. Forcing reload. Sun Sep 1 11:08:55 2019 -> Reading databases from /var/lib/clamav Sun Sep 1 11:11:01 2019 -> Database correctly reloaded (6301816 signatures) Sun Sep 1 11:11:02 2019 -> Reading databases from /var/lib/clamav Sun Sep 1 11:13:07 2019 -> Database correctly reloaded (6301816 signatures) Sun Sep 1 11:23:07 2019 -> SelfCheck: Database status OK. Sun Sep 1 11:33:08 2019 -> SelfCheck: Database status OK. Sun Sep 1 11:46:32 2019 -> SelfCheck: Database status OK. Sun Sep 1 11:56:32 2019 -> SelfCheck: Database status OK. Sun Sep 1 12:10:41 2019 -> SelfCheck: Database status OK. So only when in fact real updates come through with freshclam, clamd, running its own check cycle of 10 to 15 minutes, will do the two minute long reload. > > >On 06/09/2019 11:31, G.W. Haywood wrote: > >>That's very useful, thanks. Can you compare the costs of running > >>them for us? > > On 06.09.19 11:54, Reio Remma via clamav-users wrote: > >I suspect the i9-9900 is cheaper to actually run than the old > >whichever Core is in the mail server. :D > > I think that virtual/cloud server has to be cheaper than powe
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 06/09/2019 12:00, Matus UHLAR - fantomas wrote: On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep 6 09:41:06 mail clamd[31441]: Reading databases from /var/lib/clamav Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded (10741767 ... Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 ... Fri Sep 6 08:49:08 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 08:50:18 2019 -> Database correctly reloaded (8830356 signatures) Fri Sep 6 09:48:25 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 09:49:49 2019 -> Database correctly reloaded (8830677 signatures) Fri Sep 6 10:47:36 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 10:48:53 2019 -> Database correctly reloaded (8830954 signatures) average ~1:20 on X3440 CPU (10 years old). On 06/09/2019 11:31, G.W. Haywood wrote: That's very useful, thanks. Can you compare the costs of running them for us? On 06.09.19 11:54, Reio Remma via clamav-users wrote: I suspect the i9-9900 is cheaper to actually run than the old whichever Core is in the mail server. :D I think that virtual/cloud server has to be cheaper than power usage of the existing server (plus housing, if you pay for that one). (Un)fortunately, we're in a building with practically free electricity due to some management error. Otherwise I would have made a case to upgrade the server long ago based on power usage alone. :) ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep 6 09:41:06 mail clamd[31441]: Reading databases from /var/lib/clamav Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded (10741767 ... Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 ... Fri Sep 6 08:49:08 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 08:50:18 2019 -> Database correctly reloaded (8830356 signatures) Fri Sep 6 09:48:25 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 09:49:49 2019 -> Database correctly reloaded (8830677 signatures) Fri Sep 6 10:47:36 2019 -> Reading databases from /var/lib/clamav Fri Sep 6 10:48:53 2019 -> Database correctly reloaded (8830954 signatures) average ~1:20 on X3440 CPU (10 years old). On 06/09/2019 11:31, G.W. Haywood wrote: That's very useful, thanks. Can you compare the costs of running them for us? On 06.09.19 11:54, Reio Remma via clamav-users wrote: I suspect the i9-9900 is cheaper to actually run than the old whichever Core is in the mail server. :D I think that virtual/cloud server has to be cheaper than power usage of the existing server (plus housing, if you pay for that one). -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Linux - It's now safe to turn on your computer. Linux - Teraz mozete pocitac bez obav zapnut. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 06/09/2019 11:31, G.W. Haywood wrote: Hi there, On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep 6 09:41:06 mail clamd[31441]: Reading databases from /var/lib/clamav Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded (10741767 ... Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 ... That's very useful, thanks. Can you compare the costs of running them for us? I suspect the i9-9900 is cheaper to actually run than the old whichever Core is in the mail server. :D Reio ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep 6 09:41:06 mail clamd[31441]: Reading databases from /var/lib/clamav Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded (10741767 ... Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 ... That's very useful, thanks. Can you compare the costs of running them for us? -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 04/09/2019 23:44, Micah Snyder (micasnyd) via clamav-users wrote: The database load process reads signatures and uses the data to populate a couple of pseudo-tries (https://en.wikipedia.org/wiki/Trie). The tries themselves could only be modified by a single thread at a time, with a mutex around each trie. There might be some performance to be gained by using multiple threads. I'm not certain. Definitely a bunch of thread safety code would need to be written. I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep 6 09:41:06 mail clamd[31441]: Reading databases from /var/lib/clamav Sep 6 09:44:05 mail clamd[31441]: Database correctly reloaded (10741767 signatures) Sep 6 09:56:43 vm clamd[2108]: Reading databases from /var/lib/clamav Sep 6 09:57:17 vm clamd[2108]: Database correctly reloaded (10742128 signatures) Reio ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-09-05 11:35, schrieb G.W. Haywood via clamav-users: It seems that the two hour loading is hardcoded in the daemon. No. There are two ways to trigger reloading the databases. One is to set the 'SelfCheck' interval. The other is to send a 'RELOAD' command on the port or socket on which the daemon is listening. For example if the daemon is listening on TCP port 127.0.0.1:3311 manually I might do this at a shell prompt: $ /bin/echo 'RELOAD' | /bin/nc localhost 3311 | /usr/bin/logger -p mail.debug 2>&1 The 'SelfCheck' interval tells the daemon to reload the databases only if something has changed: That's interesting, I was able to find out where it comes from. I could change the value in a config, that is not part of clamav itself. Thanks for making that clear. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Thursday 05 September 2019, Thomas Barth via clamav-users wrote: > It seems that the two hour loading is hardcoded in the daemon. You can use freshclam without "-d" option. You can stop freshclam daemon and create /etc/cron.d/freshclam with 22 1 * * * root/usr/bin/freshclam --quiet --daemon-notify 22 3 * * * root/usr/bin/freshclam --quiet --daemon-notify 22 5 * * * root/usr/bin/freshclam --quiet --daemon-notify 22 8 * * * root/usr/bin/freshclam --quiet --daemon-notify 22 12 * * * root/usr/bin/freshclam --quiet --daemon-notify for example. -- Regards, Sergey ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Thu, 5 Sep 2019, Thomas Barth via clamav-users wrote: freshclam just downloads the standard databases to keep them fresh. In /etc/clamav/freshclam.conf you can set the check interval. That s ok. # Check for new database 24 times a day Checks 24 Good so far. But it s /usr/sbin/clamd who loads the databases into memory. Yes. In /etc/clamav/clamd.conf there should be a value of 12 for an every two hour load, right? No. It seems that the two hour loading is hardcoded in the daemon. No. There are two ways to trigger reloading the databases. One is to set the 'SelfCheck' interval. The other is to send a 'RELOAD' command on the port or socket on which the daemon is listening. For example if the daemon is listening on TCP port 127.0.0.1:3311 manually I might do this at a shell prompt: $ /bin/echo 'RELOAD' | /bin/nc localhost 3311 | /usr/bin/logger -p mail.debug 2>&1 The 'SelfCheck' interval tells the daemon to reload the databases only if something has changed: mail6:~$ >>> grep 'SelfCheck' /var/log/mail.debug ... Sep 4 02:15:12 mail6 clamd[5479]: SelfCheck: Database status OK. Sep 4 03:37:14 mail6 clamd[7689]: SelfCheck: Database status OK. Sep 4 05:02:02 mail6 clamd[5479]: SelfCheck: Database status OK. Sep 4 06:24:14 mail6 clamd[7689]: SelfCheck: Database status OK. Sep 4 07:49:13 mail6 clamd[5479]: SelfCheck: Database status OK. Sep 4 09:11:11 mail6 clamd[7689]: SelfCheck: Database modification detected. Forcing reload. Sep 4 10:36:12 mail6 clamd[5479]: SelfCheck: Database modification detected. Forcing reload. Sep 4 12:03:14 mail6 clamd[7689]: SelfCheck: Database modification detected. Forcing reload. Sep 4 13:27:12 mail6 clamd[5479]: SelfCheck: Database status OK. Sep 4 14:54:15 mail6 clamd[7689]: SelfCheck: Database status OK. Sep 4 16:14:12 mail6 clamd[5479]: SelfCheck: Database status OK. Sep 4 17:41:14 mail6 clamd[7689]: SelfCheck: Database status OK. Sep 4 19:01:12 mail6 clamd[5479]: SelfCheck: Database status OK. Sep 4 20:28:14 mail6 clamd[7689]: SelfCheck: Database status OK. Sep 4 21:48:12 mail6 clamd[5479]: SelfCheck: Database status OK. Sep 4 23:15:14 mail6 clamd[7689]: SelfCheck: Database modification detected. Forcing reload. Sep 5 00:35:12 mail6 clamd[5479]: SelfCheck: Database modification detected. Forcing reload. Sep 5 02:07:14 mail6 clamd[7689]: SelfCheck: Database modification detected. Forcing reload. Sep 5 03:26:12 mail6 clamd[5479]: SelfCheck: Database modification detected. Forcing reload. Sep 5 04:59:14 mail6 clamd[7689]: SelfCheck: Database status OK. Sep 5 06:17:12 mail6 clamd[5479]: SelfCheck: Database status OK. Sep 5 07:46:14 mail6 clamd[7689]: SelfCheck: Database status OK. Sep 5 09:04:12 mail6 clamd[5479]: SelfCheck: Database modification detected. Forcing reload. I PING the daemons every minute. I've patched the, er, patched daemon also to reply in lower case to PING commands, so that I can see which one replies when. Here's my *unpatched* daemon reloading this morning: Sep 5 09:02:12 mail6 root: PONG Sep 5 09:02:14 mail6 root: pong Sep 5 09:03:12 mail6 root: PONG Sep 5 09:03:14 mail6 root: pong Sep 5 09:04:12 mail6 clamd[5479]: SelfCheck: Database modification detected. Forcing reload. Sep 5 09:04:14 mail6 clamd[5479]: Reading databases from /etc/mail/clamav Sep 5 09:04:14 mail6 root: pong Sep 5 09:05:14 mail6 root: pong Sep 5 09:06:14 mail6 root: pong Sep 5 09:07:14 mail6 root: pong Sep 5 09:07:59 mail6 clamd[5479]: Database correctly reloaded (8869225 signatures) Sep 5 09:05:12 mail6 root: PONG Sep 5 09:06:12 mail6 root: PONG Sep 5 09:04:12 mail6 root: PONG Sep 5 09:07:12 mail6 root: PONG Sep 5 09:08:12 mail6 root: PONG Sep 5 09:08:14 mail6 root: pong Sep 5 09:09:12 mail6 root: PONG Sep 5 09:09:14 mail6 root: pong Sep 5 09:10:12 mail6 root: PONG Here's the *patched daemon reloading: Sep 5 02:06:12 mail6 root: PONG Sep 5 02:06:14 mail6 root: pong Sep 5 02:07:12 mail6 root: PONG Sep 5 02:07:14 mail6 clamd[7689]: SelfCheck: Database modification detected. Forcing reload. Sep 5 02:07:14 mail6 clamd[7689]: Reading databases from /etc/mail/clamav Sep 5 02:07:14 mail6 root: pong Sep 5 02:08:12 mail6 root: PONG Sep 5 02:08:14 mail6 root: pong Sep 5 02:09:12 mail6 root: PONG Sep 5 02:09:14 mail6 root: pong Sep 5 02:10:12 mail6 root: PONG Sep 5 02:10:14 mail6 root: pong Sep 5 02:11:12 mail6 root: PONG Sep 5 02:11:14 mail6 root: pong Sep 5 02:11:35 mail6 clamd[7689]: Database correctly reloaded (8871522 signatures) Sep 5 02:12:12 mail6 root: PONG Sep 5 02:12:14 mail6 root: pong Sep 5 02:13:12 mail6 root: PONG Sep 5 02:13:14 mail6 root: pong Sep 5 02:14:12 mail6 root: PONG Sep 5 02:14:14 mail6 root: pong See the difference? The patched daemon does what you want. The unpatched one doesn't. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-09-05 09:14, schrieb Sergey: On Thursday 05 September 2019, Thomas Barth via clamav-users wrote: Please, where can I change the interval value or times for loading the databases? You can run freshclam by cron for example. ps aux | grep clam clamav 439 0.0 0.0 51152 11360 ?Ss Aug12 2:37 /usr/bin/freshclam -d --foreground=true clamav8522 2.6 8.3 1727312 1378476 ? Ssl Sep04 38:21 /usr/sbin/clamd --foreground=true freshclam just downloads the standard databases to keep them fresh. In /etc/clamav/freshclam.conf you can set the check interval. That s ok. # Check for new database 24 times a day Checks 24 But it s /usr/sbin/clamd who loads the databases into memory. In /etc/clamav/clamd.conf there should be a value of 12 for an every two hour load, right? When I look for the value 12 I only can find the variable "MaxThreads 12" It seems that the two hour loading is hardcoded in the daemon. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Thursday 05 September 2019, Thomas Barth via clamav-users wrote: > Please, where can I change the interval value or times for loading > the databases? You can run freshclam by cron for example. -- Regards, Sergey ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-09-05 00:30, schrieb G.W. Haywood via clamav-users: The database load times are a couple of orders of magnitude shorter than the database update periods. It makes no sense to try to make the load times shorter when they can already be done by a separate thread, while scanning continues, if necessary, in another thread. Yes, for me this is a big bug. It even ruins my mailserver. For legal reasons I ve set up my mailservers to scan emails in realtime for years. No store & forward! It refuses bad emails during a connection. Two weeks ago I installed mailservers with new debian 10. On these new servers Clamav reloads the database every two hours. It takes up to two minutes, clamav blockes everything and my boss has the talent to send an email whenever the databases are loaded. He got already upset about his email client getting a timeout. On my older servers with debian 9 Clamav loaded the databases with fewer signatures once a day only! Ok, it s cool to have better security now. But I would like to disable the loadings during normal business time. Please, where can I change the interval value or times for loading the databases? ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On 9/4/19, 1:40 PM, Thomas Barth via wrote: > Why not using half of the cores to also reduce the loading time? Many > years ago when I used eMule for downloading big files, I was so > fascinated by the download mechanism: one big file, many download > sources to get the file together piece by piece. And it didn't have to > follow any order. That would be fun to programm for loading the > databases, am I right? :-) You might be right that it might be fun, but torrents are completely irrelevant to this issue, and they exist elsewhere to sove a problem which does not exist here. On Wed, 4 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote: ... There might be some performance to be gained by using multiple threads. I'm not certain. Definitely a bunch of thread safety code would need to be written. The database load times are a couple of orders of magnitude shorter than the database update periods. It makes no sense to try to make the load times shorter when they can already be done by a separate thread, while scanning continues, if necessary, in another thread. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
The database load process reads signatures and uses the data to populate a couple of pseudo-tries (https://en.wikipedia.org/wiki/Trie). The tries themselves could only be modified by a single thread at a time, with a mutex around each trie. There might be some performance to be gained by using multiple threads. I'm not certain. Definitely a bunch of thread safety code would need to be written. -Micah On 9/4/19, 1:40 PM, "clamav-users on behalf of Thomas Barth via clamav-users" wrote: Am 2019-09-01 19:30, schrieb Joel Esler (jesler) via clamav-users: > Alright. I think we’ve beat the proverbial dead horse here. The devs > know this is a request and they will get it into their dev queue for > examination. I saw that clamd use just one core at a time to load the databases. top - 16:09:43 up 23:33, 2 users, load average: 0.47, 0.13, 0.04 Tasks: 176 total, 2 running, 174 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st %Cpu1 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 : 85.4 us, 6.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 7.8 st %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 16042.2 total, 14207.5 free,813.4 used, 1021.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 14921.4 avail Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 455 clamav20 0 469352 207432 10840 R 100.0 1.3 30:33.51 /usr/sbin/clamd --foreground=true 22861 root 20 0 11316 3648 3108 R 2.0 0.0 0:24.91 top Always 2 minute loading time Wed Sep 4 16:09:17 2019 -> Reading databases from /var/lib/clamav Wed Sep 4 16:11:24 2019 -> Database correctly reloaded (10966440 signatures) Why not using half of the cores to also reduce the loading time? Many years ago when I used eMule for downloading big files, I was so fascinated by the download mechanism: one big file, many download sources to get the file together piece by piece. And it didn't have to follow any order. That would be fun to programm for loading the databases, am I right? :-) ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-09-01 19:30, schrieb Joel Esler (jesler) via clamav-users: Alright. I think we’ve beat the proverbial dead horse here. The devs know this is a request and they will get it into their dev queue for examination. I saw that clamd use just one core at a time to load the databases. top - 16:09:43 up 23:33, 2 users, load average: 0.47, 0.13, 0.04 Tasks: 176 total, 2 running, 174 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st %Cpu1 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 : 85.4 us, 6.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 7.8 st %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 16042.2 total, 14207.5 free,813.4 used, 1021.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 14921.4 avail Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 455 clamav20 0 469352 207432 10840 R 100.0 1.3 30:33.51 /usr/sbin/clamd --foreground=true 22861 root 20 0 11316 3648 3108 R 2.0 0.0 0:24.91 top Always 2 minute loading time Wed Sep 4 16:09:17 2019 -> Reading databases from /var/lib/clamav Wed Sep 4 16:11:24 2019 -> Database correctly reloaded (10966440 signatures) Why not using half of the cores to also reduce the loading time? Many years ago when I used eMule for downloading big files, I was so fascinated by the download mechanism: one big file, many download sources to get the file together piece by piece. And it didn't have to follow any order. That would be fun to programm for loading the databases, am I right? :-) ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-09-01 19:12, schrieb G.W. Haywood via clamav-users: Hi there, On Sun, 1 Sep 2019, Thomas Barth via clamav-users wrote: Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: That's exactly what the patch in #10979 does. ... And where can I find this patch? If you navigate to https://bugzilla.clamav.net/show_bug.cgi?id=10979 [...] Thank you for your detailed and patient explanation. I thought it s just a configure && install ;-) But in the moment, it would be too much for me to manage clamav by hand. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi Joel, On Sun, 1 Sep 2019, Joel Esler (jesler) wrote: Alright. I think we’ve beat the proverbial dead horse here. ... I don't think anybody's beating anything here Joel. Just we users, discussing, on the users' list, ways of dealing with an issue. On Sat, 31 Aug 2019, G.W. Haywood wrote: It really isn't that big a deal if you know what you're doing. You saw that part? -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Alright. I think we’ve beat the proverbial dead horse here. The devs know this is a request and they will get it into their dev queue for examination. Sent from my iPhone > On Sep 1, 2019, at 13:21, G.W. Haywood via clamav-users > wrote: > > Hi there, > >> On Sun, 1 Sep 2019, Thomas Barth via clamav-users wrote: >> >> Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: >>> That's exactly what the patch in #10979 does. ... >> >> And where can I find this patch? > > If you navigate to > > https://bugzilla.clamav.net/show_bug.cgi?id=10979 > > and then down to "Comment 2" (dated 2016-11-28 12:16:52 EST) you will > see a link "attachment 7196". This is a modification to the original > #10979 patch. If you navigate to that link you will see a page which > gives a representation of the patch 'diff'. Near the top of that page > there is a link "Raw Unified", which takes you to the raw unified diff > text which is here: > > https://bugzilla.clamav.net/attachment.cgi?id=7196=diff=patch==1=raw > > You might be able to use this as input to 'patch' but I didn't try it, > I did not expect it to work well on code which is years younger than > that on which the patch is based. Instead, I applied the patch by > hand with an editor. It was tedious but not difficult. Even if you > do not believe that you can trust my patched files (which I think is a > perfectly reasonable belief:) I should be happy to mail the patched > files to you so that you can compare the results of patching to give > you some confidence that it will work. > >> ... what happens if I update my system (# aptitude update && >> aptitude safe-upgrade) and a new verson of clamav is being >> installed. Do I always have to repatch clamav? > > If you want to use this patch you must compile and install ClamAV from > the sources distributed on the clamav.net Website. You cannot use the > package management system of any Operating System (OS) distribution to > install any version of the ClamAV package(s) from the OS distribution. > Of course you could create your own package from the patched sources, > and then use the package management system to install your own package. > Many administrators do that when they have large numbers of machines > to be installed but they have some reason to avoid using the packages > produced by the OS publisher. If 'upstream' produces a new version of > the package which (still) does not contain the patch then yes, you do > have to re-apply the patch. > > Your package manager will probably set up ClamAV in a way which is > very different from the way it is set up after building from source, > e.g. using directory paths like /usr/bin and /usr/sbin instead of > /usr/local/bin, /usr/local/sbin etc. - here are some samples from a > machine with both kinds of package installed: > > mail6:~$ >>> l /usr/sbin/clam* > -rwxr-xr-x 1 root root 223296 Apr 15 22:12 /usr/sbin/clamd > -rwxr-xr-x 1 root root 233424 Apr 15 22:12 /usr/sbin/clamav-milter > mail6:~$ >>> l /usr/local/sbin/clam* > -rwxr-xr-x 1 root staff 581080 Aug 21 18:43 /usr/local/sbin/clamd > -rwxr-xr-x 1 root staff 581368 Aug 22 14:33 /usr/local/sbin/clamd_patched > mail6:~$ >>> l /usr/bin/freshclam > -rwxr-xr-x 1 root root 202816 Apr 15 22:12 /usr/bin/freshclam > mail6:~$ >>> l /usr/local/bin/freshclam > -rwxr-xr-x 1 root staff 442616 Aug 22 14:33 /usr/local/bin/freshclam > > Note that there are THREE versions of clamd on this machine - the OS > distribution version and two versions built from source. The versions > built from source are the two which are currently running on the machine: > > mail6:~$ >>> top -n1 -b -u clamav > top - 18:04:21 up 9 days, 1:49, 9 users, load average: 0.11, 0.33, 0.29 > Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie > %Cpu(s): 2.1 us, 0.5 sy, 0.1 ni, 92.5 id, 0.3 wa, 0.0 hi, 4.4 si, 0.0 > st > KiB Mem: 16469180 total, 15243004 used, 1226176 free, 232408 buffers > KiB Swap: 3212284 total,0 used, 3212284 free. 11851656 cached Mem > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 3846 clamav20 0 61220 5644 4568 S 0.0 0.0 4:07.37 freshclam > 5479 clamav20 0 1430760 1.058g 4604 S 0.0 6.7 115:21.15 clamd > 7689 clamav20 0 1490600 1.061g 4656 S 0.0 6.8 123:10.10 > clamd_patched > > There will be other path differences too, for configuration and data > file stores. If you do something like this then you need to make sure > that you're running the right binaries, and that the binaries will use > the right configurations and libraries. If you aren't sure you can do > that then it would be best to uninstall and *purge* the OS versions of > the packages before you install the package from source. This applies > not just to ClamAV, but to any package where there may be conflicts of > this kind. > > HTH > > -- > > 73, > Ged. > > ___ > > clamav-users
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Sun, 1 Sep 2019, Thomas Barth via clamav-users wrote: Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: That's exactly what the patch in #10979 does. ... And where can I find this patch? If you navigate to https://bugzilla.clamav.net/show_bug.cgi?id=10979 and then down to "Comment 2" (dated 2016-11-28 12:16:52 EST) you will see a link "attachment 7196". This is a modification to the original #10979 patch. If you navigate to that link you will see a page which gives a representation of the patch 'diff'. Near the top of that page there is a link "Raw Unified", which takes you to the raw unified diff text which is here: https://bugzilla.clamav.net/attachment.cgi?id=7196=diff=patch==1=raw You might be able to use this as input to 'patch' but I didn't try it, I did not expect it to work well on code which is years younger than that on which the patch is based. Instead, I applied the patch by hand with an editor. It was tedious but not difficult. Even if you do not believe that you can trust my patched files (which I think is a perfectly reasonable belief:) I should be happy to mail the patched files to you so that you can compare the results of patching to give you some confidence that it will work. ... what happens if I update my system (# aptitude update && aptitude safe-upgrade) and a new verson of clamav is being installed. Do I always have to repatch clamav? If you want to use this patch you must compile and install ClamAV from the sources distributed on the clamav.net Website. You cannot use the package management system of any Operating System (OS) distribution to install any version of the ClamAV package(s) from the OS distribution. Of course you could create your own package from the patched sources, and then use the package management system to install your own package. Many administrators do that when they have large numbers of machines to be installed but they have some reason to avoid using the packages produced by the OS publisher. If 'upstream' produces a new version of the package which (still) does not contain the patch then yes, you do have to re-apply the patch. Your package manager will probably set up ClamAV in a way which is very different from the way it is set up after building from source, e.g. using directory paths like /usr/bin and /usr/sbin instead of /usr/local/bin, /usr/local/sbin etc. - here are some samples from a machine with both kinds of package installed: mail6:~$ >>> l /usr/sbin/clam* -rwxr-xr-x 1 root root 223296 Apr 15 22:12 /usr/sbin/clamd -rwxr-xr-x 1 root root 233424 Apr 15 22:12 /usr/sbin/clamav-milter mail6:~$ >>> l /usr/local/sbin/clam* -rwxr-xr-x 1 root staff 581080 Aug 21 18:43 /usr/local/sbin/clamd -rwxr-xr-x 1 root staff 581368 Aug 22 14:33 /usr/local/sbin/clamd_patched mail6:~$ >>> l /usr/bin/freshclam -rwxr-xr-x 1 root root 202816 Apr 15 22:12 /usr/bin/freshclam mail6:~$ >>> l /usr/local/bin/freshclam -rwxr-xr-x 1 root staff 442616 Aug 22 14:33 /usr/local/bin/freshclam Note that there are THREE versions of clamd on this machine - the OS distribution version and two versions built from source. The versions built from source are the two which are currently running on the machine: mail6:~$ >>> top -n1 -b -u clamav top - 18:04:21 up 9 days, 1:49, 9 users, load average: 0.11, 0.33, 0.29 Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie %Cpu(s): 2.1 us, 0.5 sy, 0.1 ni, 92.5 id, 0.3 wa, 0.0 hi, 4.4 si, 0.0 st KiB Mem: 16469180 total, 15243004 used, 1226176 free, 232408 buffers KiB Swap: 3212284 total,0 used, 3212284 free. 11851656 cached Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 3846 clamav20 0 61220 5644 4568 S 0.0 0.0 4:07.37 freshclam 5479 clamav20 0 1430760 1.058g 4604 S 0.0 6.7 115:21.15 clamd 7689 clamav20 0 1490600 1.061g 4656 S 0.0 6.8 123:10.10 clamd_patched There will be other path differences too, for configuration and data file stores. If you do something like this then you need to make sure that you're running the right binaries, and that the binaries will use the right configurations and libraries. If you aren't sure you can do that then it would be best to uninstall and *purge* the OS versions of the packages before you install the package from source. This applies not just to ClamAV, but to any package where there may be conflicts of this kind. HTH -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: That's exactly what the patch in #10979 does. Unfortunately, although as I've said it's simple enough to apply the patch, it's by no means a simple patch and it would greatly benefit from some serious testing by the community - especially by people who see higher volumes of mail than I do. On 01.09.19 18:08, Thomas Barth via clamav-users wrote: And where can I find this patch? It s not on the download page (https://www.clamav.net/downloads), so it s not official. I would like to test it on my private server first, just to see if I get it work. And what happens if I update my system (# aptitude update && aptitude safe-upgrade) and a new verson of clamav is being installed. Do I always have to repatch clamav? it's attached to the bugreport -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Saving Private Ryan... Private Ryan exists. Overwrite? (Y/N) ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: That's exactly what the patch in #10979 does. Unfortunately, although as I've said it's simple enough to apply the patch, it's by no means a simple patch and it would greatly benefit from some serious testing by the community - especially by people who see higher volumes of mail than I do. And where can I find this patch? It s not on the download page (https://www.clamav.net/downloads), so it s not official. I would like to test it on my private server first, just to see if I get it work. And what happens if I update my system (# aptitude update && aptitude safe-upgrade) and a new verson of clamav is being installed. Do I always have to repatch clamav? ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Sat, 31 Aug 2019, J.R. via clamav-users wrote: ... I wouldn't call the current design a "bug"... It works as intended. +1 However it would be nice if a fresh DB could be parsed & loaded, then swapped, to prevent service interruption. That's exactly what the patch in #10979 does. Unfortunately, although as I've said it's simple enough to apply the patch, it's by no means a simple patch and it would greatly benefit from some serious testing by the community - especially by people who see higher volumes of mail than I do. Perhaps we should call it "crowd-sourcing"? Would that be better? :) -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
> Normally postfix gets a response after 3 secondes. > > In the clamav.log I see at the same time, that reloading the database > takes up to two minutes. Yes, reloading the DB can take some time depending on which signature DBs you are using. I can't speak for postfix (I run sendmail), but on my server if it can't run the AV scan, then it simply tempfails the email and the remote server (should) try later. I wouldn't call the current design a "bug"... It works as intended. However it would be nice if a fresh DB could be parsed & loaded, then swapped, to prevent service interruption. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On August 31, 2019 4:32:00 PM UTC, Henrik K wrote: >On Sat, Aug 31, 2019 at 12:21:11PM -0400, Scott Kitterman via >clamav-users wrote: >> >> Not to put too fine a point on it, but if you are unhappy with the >service you >> are receiving, you should switch to a different vendor. I suspect >it's >> unlikely you'll get the same value for money elsewhere. > >Does this worn cliche really need posting? :-) > >But hey, I'm just participating in the community.. sometimes things >just >need a bit of nudging. I wouldn't even continue to nag about it, if >this >was a basic volunteer project. But we are talking about a security >company >that should be proud of it's code. There's no problem with nudging, but being nasty about isn't appropriate. I'll confess that I aimed a comment at you that should have been pointed at the OP. Sorry about that. I've been maintaining clamav packages for over a decade through three different companies owning the project and overall I think the Talos/Cisco people are doing a pretty good job. They are generally responsive. Also, in terms of being proud of their code, you should compare the code quality when Sourcefire bought clamav to the current code base. It's night and day different. Also the rate of security bugs seems to have dropped off (not to mention we actually get bugfix releases now). So yeah, they could do better, but looking back, I think they're doing pretty good. In this case, the 'worn cliche' is important, because unlike lots of other FOSS projects, this one doesn't have non-proprietary alternatives, so it'd be pretty awful if the community were to convince Talos that publishing it was more trouble than it was worth. Scott K ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Sat, Aug 31, 2019 at 12:21:11PM -0400, Scott Kitterman via clamav-users wrote: > > Not to put too fine a point on it, but if you are unhappy with the service > you > are receiving, you should switch to a different vendor. I suspect it's > unlikely you'll get the same value for money elsewhere. Does this worn cliche really need posting? :-) But hey, I'm just participating in the community.. sometimes things just need a bit of nudging. I wouldn't even continue to nag about it, if this was a basic volunteer project. But we are talking about a security company that should be proud of it's code. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Sat, 31 Aug 2019, Henrik K wrote: On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users wrote: The final responsibility of implementing and testing the issue is still that of the ClamAV team. Agreed. You are really making this much more complex and "scary" issue than it is. No, I don't think I am. How much experience do you have of writing thread-safe code in C? -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Saturday, August 31, 2019 12:04:36 PM EDT Henrik K wrote: > On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users wrote: > > More testing, by people prepared to chip in some effort instead of > > complaining about something that they get for free, would be great. > > The final responsibility of implementing and testing the issue is still that > of the ClamAV team. > > You are really making this much more complex and "scary" issue than it is. > New features and major versions have been constantly released these past > years. Just because someone in the bug had a random issue with patch that > wasn't even analyzed by devs, doesn't mean it will "break millions of > systems" - especially if it isn't enabled by default (which is wise, since > it would need more memory). It's simply a matter of willing to check and > implement it. Not to put too fine a point on it, but if you are unhappy with the service you are receiving, you should switch to a different vendor. I suspect it's unlikely you'll get the same value for money elsewhere. Scott K ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users wrote: > > More testing, by people prepared to chip in some effort instead of > complaining about something that they get for free, would be great. The final responsibility of implementing and testing the issue is still that of the ClamAV team. You are really making this much more complex and "scary" issue than it is. New features and major versions have been constantly released these past years. Just because someone in the bug had a random issue with patch that wasn't even analyzed by devs, doesn't mean it will "break millions of systems" - especially if it isn't enabled by default (which is wise, since it would need more memory). It's simply a matter of willing to check and implement it. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Sat, 31 Aug 2019, Henrik K wrote: ... If I encountered a bug like that on some project that I'm maintaining, I would be shamed not to rapidly fix it. If you called it a limitation I could agree, but I guess it's working as designed. I'd call it an issue rather than a fault in the software. If there _are_ bugs in this issue they're in the patch for it, which may be why, AFAICT, I'm one of only about three people on the planet who are actually running it. More testing, by people prepared to chip in some effort instead of complaining about something that they get for free, would be great. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Sat, 31 Aug 2019, Henrik K wrote: On Sat, Aug 31, 2019, G.W. Haywood via clamav-users wrote: Well not quite nothing, since you can download the source, apply the patch, and rebuild ClamAV. Sure but it's not reality for majority of users.. While it's good that people try it out, I doubt if would take long for a dev to verify the patch carefully and implement boolean for it's use. But I guess new features pay more than having a robust engine. It's not quite as simple as that. This software has to run reliably on millions of systems with thousands of combinations and permutations of configurations. It's doing that right now. There've occasionally been examples of a change made perhaps a little too hastily which gave grief to many users and rise to a lot of spleen-venting on the users' mailing list. It would be a brave decision, in the face of the valid concerns noted in #10979, to release a new version, world-wide, for production use, which contains the patch that I'm running now merely as an experiment with my eyes wide open on a server that crashed four times this month because I'm also working on some netfilter stuff. This is a community effort. If you're familiar with C it isn't at all difficult to apply the patch, and I'd be happy to mail the two patched files (56kBytes in total) to anyone who didn't feel up to applying the patches themselves. Then, if you felt brave enough, it would _almost_ be as simple as ./configure && make && sudo make install to build and install it. Incidentally I'm a Sendmail dinosaur, and the default timeouts appear to be longer for Sendmail than they are for Postfix. I'm sure it's easy to make them longer for Postfix; then this issue would, if not disappear, at least more or less be transparent. It really isn't that big a deal if you know what you're doing. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Sat, Aug 31, 2019 at 11:18:00AM -0400, Michael Orlitzky via clamav-users wrote: > > Micah took the time to answer a question and provide a status update. > It's counterproductive to shame people for being honest. It's perfectly fine to shame a corporation for doing seemingly strange things. Micah etc are paid developers and not volunteers maintaining some stale Open Source thingy. Well atleast I hope they are not.. An existing patch has existed for 5 years, so I'm pretty interested in hearing why such a basic and important feature is still not implemented. Only thing that comes to mind is that the developers don't even actually use ClamAV personally, or the use is so marginal that they don't even encounter this problem. If I encountered a bug like that on some project that I'm maintaining, I would be shamed not to rapidly fix it. But perhaps it's the organization to blame. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On 8/31/19 11:00 AM, Thomas Barth via clamav-users wrote: > > Realy bad attitude of developers! Micah took the time to answer a question and provide a status update. It's counterproductive to shame people for being honest. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
On Sat, Aug 31, 2019 at 03:55:30PM +0100, G.W. Haywood via clamav-users wrote: > > Well not quite nothing, since you can download the source, apply the > patch, and rebuild ClamAV. Sure but it's not reality for majority of users.. While it's good that people try it out, I doubt if would take long for a dev to verify the patch carefully and implement boolean for it's use. But I guess new features pay more than having a robust engine. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Am 2019-08-31 16:32, schrieb Henrik K: The reload bug has been known for years, even has a ready patch. Wow, this is a masterpiece ignoring a problem for years :) Thanks for pointing to the bugthread. But nothing you can do about it, ClamAV devs have a mind of their own. Micah Snyder 2019-08-22 14:38:59 EDT "We are not actively working on this, though it is on our list." Realy bad attitude of developers! Fixing errors and problems must always have highest priority before developing new things! Atleast servers in your scenario will (hopefully) retry sending. Not the spammer. On Sat, Aug 31, 2019 at 04:25:05PM +0200, Thomas Barth via clamav-users wrote: Hallo Mailinglist, sometimes I get in Postfix the error messages "451 4.3.0 Error: queue file write error". There is a warning timeout talking to localhost:10024 (Amavis) Aug 31 14:14:19 mx2 postfix/smtpd[15861]: connect from unknown[177.37.96.254] Aug 31 14:14:20 mx2 postfix/smtpd[15861]: NOQUEUE: client=unknown[177.37.96.254] Aug 31 14:16:02 mx2 postfix/smtpd[15861]: warning: timeout talking to proxy localhost:10024 Aug 31 14:16:02 mx2 postfix/smtpd[15861]: proxy-reject: END-OF-MESSAGE: 451 4.3.0 Error: queue file write error; from= to= proto=ESMTP helo= Aug 31 14:16:02 mx2 postfix/smtpd[15861]: disconnect from unknown[177.37.96.254] ehlo=1 mail=1 rcpt=1 data=0/1 commands=3/4 (Not hiding the from address, it s used by a spammer :)) Normally postfix gets a response after 3 secondes. In the clamav.log I see at the same time, that reloading the database takes up to two minutes. /var/log/clamav/clamav.log Sat Aug 31 14:14:15 2019 -> Database correctly reloaded (10971844 signatures) Sat Aug 31 14:14:15 2019 -> Reading databases from /var/lib/clamav Sat Aug 31 14:14:15 2019 -> /var/lib/amavis/tmp/amavis-20190831T125532-12347-lWbaS7Ci/parts/p001: Sanesecurity.Scam.12584.UNOFFICIAL(:6617) FOUND Sat Aug 31 14:16:13 2019 -> Database correctly reloaded (10971844 signatures) Sat Aug 31 14:16:13 2019 -> /var/lib/amavis/tmp/amavis-20190831T120830-10930-zSEWR54L/parts/p001: Sanesecurity.Scam.12559.UNOFFICIAL(:6449) FOUND Is reloading a database blocking the e-Mail scanning? So how can I boost this process? It's a virtual server with 100% ssd and 6 cores (Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz) and Debian Buster. Best regards, Thomas Barth ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
Hi there, On Sat, 31 Aug 2019, Henrik K wrote: The reload bug has been known for years, even has a ready patch. https://bugzilla.clamav.net/show_bug.cgi?id=10979 But nothing you can do about it... Well not quite nothing, since you can download the source, apply the patch, and rebuild ClamAV. At the moment I'm scanning mail with two copies of clamd, one patched and one plain vanilla. Despite some concerns about the reliability in #10979, which is why I'm running an unpatched copy as well, the patched version seems to be holding up - at least at fairly low mail volumes. This is with my own Perl milter, see my recent post on the dev list describing it. If anyone wants to try it they're more than welcome. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?
The reload bug has been known for years, even has a ready patch. https://bugzilla.clamav.net/show_bug.cgi?id=10979 But nothing you can do about it, ClamAV devs have a mind of their own. Atleast servers in your scenario will (hopefully) retry sending. On Sat, Aug 31, 2019 at 04:25:05PM +0200, Thomas Barth via clamav-users wrote: > Hallo Mailinglist, > > sometimes I get in Postfix the error messages "451 4.3.0 Error: queue file > write error". There is a warning timeout talking to localhost:10024 (Amavis) > > > Aug 31 14:14:19 mx2 postfix/smtpd[15861]: connect from > unknown[177.37.96.254] > Aug 31 14:14:20 mx2 postfix/smtpd[15861]: NOQUEUE: > client=unknown[177.37.96.254] > Aug 31 14:16:02 mx2 postfix/smtpd[15861]: warning: timeout talking to proxy > localhost:10024 > Aug 31 14:16:02 mx2 postfix/smtpd[15861]: proxy-reject: END-OF-MESSAGE: 451 > 4.3.0 Error: queue file write error; from= > to= proto=ESMTP helo= > Aug 31 14:16:02 mx2 postfix/smtpd[15861]: disconnect from > unknown[177.37.96.254] ehlo=1 mail=1 rcpt=1 data=0/1 commands=3/4 > > (Not hiding the from address, it s used by a spammer :)) > > Normally postfix gets a response after 3 secondes. > > In the clamav.log I see at the same time, that reloading the database takes > up to two minutes. > > /var/log/clamav/clamav.log > Sat Aug 31 14:14:15 2019 -> Database correctly reloaded (10971844 > signatures) > Sat Aug 31 14:14:15 2019 -> Reading databases from /var/lib/clamav > Sat Aug 31 14:14:15 2019 -> > /var/lib/amavis/tmp/amavis-20190831T125532-12347-lWbaS7Ci/parts/p001: > Sanesecurity.Scam.12584.UNOFFICIAL(:6617) > FOUND > Sat Aug 31 14:16:13 2019 -> Database correctly reloaded (10971844 > signatures) > Sat Aug 31 14:16:13 2019 -> > /var/lib/amavis/tmp/amavis-20190831T120830-10930-zSEWR54L/parts/p001: > Sanesecurity.Scam.12559.UNOFFICIAL(:6449) > FOUND > > Is reloading a database blocking the e-Mail scanning? So how can I boost > this process? It's a virtual server with 100% ssd and 6 cores (Intel(R) > Xeon(R) CPU E5-2630 v4 @ 2.20GHz) and Debian Buster. > > > Best regards, > Thomas Barth > > ___ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
