Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2020-02-11 Thread Reio Remma via clamav-users
On 31/10/2019 12:04, Reio Remma wrote: On 28/10/2019 12:55, Reio Remma via clamav-users wrote: On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote: Hi Micah, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote: I'm sorry, Ged... Apology accepted. :) I'm now running the development

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-10-31 Thread Reio Remma via clamav-users
On 28/10/2019 12:55, Reio Remma via clamav-users wrote: On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote: Hi Micah, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote: I'm sorry, Ged... Apology accepted. :) I'm now running the development (0.102) version of clamd, patched with Mr.

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-10-30 Thread G.W. Haywood via clamav-users
Hi Reio, On Mon, 28 Oct 2019, Reio Remma via clamav-users wrote: ... I've been running a patched 101.4 for a few weeks now and unfortunately I'm observing a memory leak from the multithreaded database reloads. I'm observing clamd memory usage going up when the new database loads ... The

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-10-28 Thread Reio Remma via clamav-users
On 14/09/2019 17:34, G.W. Haywood via clamav-users wrote: Hi Micah, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote: I'm sorry, Ged... Apology accepted. :) I'm now running the development (0.102) version of clamd, patched with Mr. Wu's patch, alongside two version 101.4 clamd daemons

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-15 Thread J.R. via clamav-users
> One thing we could do is have clamd "start" before loading the database. > That is to say that it would immediately begin listening on the unix/tcp > socket > for requests and fork into the background so as not to block the boot process. > All scan requests would then be blocked while the

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-14 Thread G.W. Haywood via clamav-users
Hi Micah, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) wrote: I'm sorry, Ged... Apology accepted. :) I'm now running the development (0.102) version of clamd, patched with Mr. Wu's patch, alongside two version 101.4 clamd daemons (an unpatched one, and one with the patch that I posted on

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-13 Thread Ralph Seichter via clamav-users
* Micah Snyder via clamav-users: > [ClamAV] would immediately begin listening on the unix/tcp socket for > requests and fork into the background so as not to block the boot > process. To me, slowing down the boot process is just the (admittedly annoying) symptom of an underlying ClamAV issue.

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-13 Thread Micah Snyder (micasnyd) via clamav-users
I'm sorry, Ged. I didn't mean to demean the work of Julius Plenz, asulfrian, or yourself. I stepped into my current position on the ClamAV team just over two years ago and in my time here there have been many tasks that have been on the backburner or ignored entirely for multiple years. We

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-13 Thread Micah Snyder (micasnyd) via clamav-users
Ged, That's a fair assessment. This is why I asked. Thanks, Micah On 9/13/19, 11:26 AM, "clamav-users on behalf of G.W. Haywood via clamav-users" wrote: Hi there, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote: > One thing we could do is have

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-13 Thread G.W. Haywood via clamav-users
Hi there, On Fri, 13 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote: One thing we could do is have clamd "start" before loading the database. That is to say that it would immediately begin listening on the unix/tcp socket for requests and fork into the background so as not to block

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-13 Thread Matus UHLAR - fantomas
On 13.09.19 14:42, Micah Snyder (micasnyd) via clamav-users wrote: One thing we could do is have clamd "start" before loading the database. That is to say that it would immediately begin listening on the unix/tcp socket for requests and fork into the background so as not to block the boot

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-13 Thread G.W. Haywood via clamav-users
Hi there, On Thu, 12 Sep 2019, Micah Snyder (micasnyd) via clamav-users wrote: https://bugzilla.clamav.net/show_bug.cgi?id=10979#c19 This patch applies to the current head of dev/0.102 ... If the development version is a step too far, the two files which I posted on September 10th implement

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-13 Thread Micah Snyder (micasnyd) via clamav-users
One thing we could do is have clamd "start" before loading the database. That is to say that it would immediately begin listening on the unix/tcp socket for requests and fork into the background so as not to block the boot process. All scan requests would then be blocked while the database

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-13 Thread Thomas Barth via clamav-users
Am 2019-09-13 00:57, schrieb Micah Snyder (micasnyd) via clamav-users: Henrik, all: Alberto Wu, a former ClamAV core developer, has very kindly provided a patch to us today that he created to perform clamd database reloads in a separate thread. I have attached his work, with some minor tweaks

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-12 Thread J.R. via clamav-users
This patch will be a very welcome addition! Oddly enough today my hosting company had an emergency and I needed to shutdown my server so it could be physically moved mid-day. The painfully slow load time of ClamAV was excruciating apparent while I was watching the console slowly go through the

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-12 Thread Micah Snyder (micasnyd) via clamav-users
Henrik, all: Alberto Wu, a former ClamAV core developer, has very kindly provided a patch to us today that he created to perform clamd database reloads in a separate thread. I have attached his work, with some minor tweaks on my part, to the relevant Bugzilla ticket, here:

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-07 Thread Thomas Barth via clamav-users
Am 2019-09-07 21:03, schrieb Robert M. Stockmann via clamav-users: Why everyone needs two minutes for this task, independent from which hardware is used, is a puzzle to me. Anyone who has the clamd .cvd files loaded on a fast SSD storage ? I also use unofficial signatures from several

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-07 Thread Al Varnell via clamav-users
To: clamav-users@lists.clamav.net <mailto:clamav-users@lists.clamav.net> >> Cc: Reio Remma mailto:r...@mrstuudio.ee>> >> Subject: Re: [clamav-users] How to boost clamav? Reloading database >>results in a talking timeout? >> >> On 07.09.2019 22:03, Robert

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-07 Thread Robert M. Stockmann via clamav-users
On Sat, 7 Sep 2019, Reio Remma via clamav-users wrote: > Date: Sat, 7 Sep 2019 22:57:08 +0300 > From: Reio Remma via clamav-users > To: clamav-users@lists.clamav.net > Cc: Reio Remma > Subject: Re: [clamav-users] How to boost clamav? Reloading database > results in

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-07 Thread Reio Remma via clamav-users
On 07.09.2019 22:03, Robert M. Stockmann via clamav-users wrote: On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep  6 09:41:06 mail clamd[31441]:

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-07 Thread Matus UHLAR - fantomas
>>On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: >>>I guess many of us are just running too old hardware. :) >>> >>>Here's a comparison between my mail server and identical config >>>running in a VM. >>> >>>Sep  6 09:41:06 mail clamd[31441]: Reading databases from >>>/var/lib/clamav

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-07 Thread Robert M. Stockmann via clamav-users
On Fri, 6 Sep 2019, Matus UHLAR - fantomas wrote: > Date: Fri, 6 Sep 2019 11:00:20 +0200 > From: Matus UHLAR - fantomas > Reply-To: ClamAV users ML > To: clamav-users@lists.clamav.net > Subject: Re: [clamav-users] How to boost clamav? Reloading database > results in

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-06 Thread Reio Remma via clamav-users
On 06/09/2019 12:00, Matus UHLAR - fantomas wrote: On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep  6 09:41:06 mail clamd[31441]: Reading

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-06 Thread Matus UHLAR - fantomas
On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep  6 09:41:06 mail clamd[31441]: Reading databases from /var/lib/clamav Sep  6 09:44:05 mail

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-06 Thread Reio Remma via clamav-users
On 06/09/2019 11:31, G.W. Haywood wrote: Hi there, On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep  6 09:41:06 mail clamd[31441]: Reading

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-06 Thread G.W. Haywood via clamav-users
Hi there, On Fri, 6 Sep 2019, Reio Remma via clamav-users wrote: I guess many of us are just running too old hardware. :) Here's a comparison between my mail server and identical config running in a VM. Sep  6 09:41:06 mail clamd[31441]: Reading databases from /var/lib/clamav Sep  6 09:44:05

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-06 Thread Reio Remma via clamav-users
On 04/09/2019 23:44, Micah Snyder (micasnyd) via clamav-users wrote: The database load process reads signatures and uses the data to populate a couple of pseudo-tries (https://en.wikipedia.org/wiki/Trie). The tries themselves could only be modified by a single thread at a time, with a mutex

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-05 Thread Thomas Barth via clamav-users
Am 2019-09-05 11:35, schrieb G.W. Haywood via clamav-users: It seems that the two hour loading is hardcoded in the daemon. No. There are two ways to trigger reloading the databases. One is to set the 'SelfCheck' interval. The other is to send a 'RELOAD' command on the port or socket on

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-05 Thread Sergey
On Thursday 05 September 2019, Thomas Barth via clamav-users wrote: > It seems that the two hour loading is hardcoded in the daemon. You can use freshclam without "-d" option. You can stop freshclam daemon and create /etc/cron.d/freshclam with 22 1 * * * root/usr/bin/freshclam

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-05 Thread G.W. Haywood via clamav-users
Hi there, On Thu, 5 Sep 2019, Thomas Barth via clamav-users wrote: freshclam just downloads the standard databases to keep them fresh. In /etc/clamav/freshclam.conf you can set the check interval. That s ok. # Check for new database 24 times a day Checks 24 Good so far. But it s

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-05 Thread Thomas Barth via clamav-users
Am 2019-09-05 09:14, schrieb Sergey: On Thursday 05 September 2019, Thomas Barth via clamav-users wrote: Please, where can I change the interval value or times for loading the databases? You can run freshclam by cron for example. ps aux | grep clam clamav 439 0.0 0.0 51152 11360 ?

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-05 Thread Sergey
On Thursday 05 September 2019, Thomas Barth via clamav-users wrote: > Please, where can I change the interval value or times for loading > the databases? You can run freshclam by cron for example. -- Regards, Sergey ___ clamav-users mailing list

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-05 Thread Thomas Barth via clamav-users
Am 2019-09-05 00:30, schrieb G.W. Haywood via clamav-users: The database load times are a couple of orders of magnitude shorter than the database update periods. It makes no sense to try to make the load times shorter when they can already be done by a separate thread, while scanning

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-04 Thread G.W. Haywood via clamav-users
Hi there, On 9/4/19, 1:40 PM, Thomas Barth via wrote: > Why not using half of the cores to also reduce the loading time? Many > years ago when I used eMule for downloading big files, I was so > fascinated by the download mechanism: one big file, many download > sources to get the file

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-04 Thread Micah Snyder (micasnyd) via clamav-users
The database load process reads signatures and uses the data to populate a couple of pseudo-tries (https://en.wikipedia.org/wiki/Trie). The tries themselves could only be modified by a single thread at a time, with a mutex around each trie. There might be some performance to be gained by

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-04 Thread Thomas Barth via clamav-users
Am 2019-09-01 19:30, schrieb Joel Esler (jesler) via clamav-users: Alright. I think we’ve beat the proverbial dead horse here. The devs know this is a request and they will get it into their dev queue for examination. I saw that clamd use just one core at a time to load the databases. top -

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-02 Thread Thomas Barth via clamav-users
Am 2019-09-01 19:12, schrieb G.W. Haywood via clamav-users: Hi there, On Sun, 1 Sep 2019, Thomas Barth via clamav-users wrote: Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: That's exactly what the patch in #10979 does. ... And where can I find this patch? If you navigate

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-01 Thread G.W. Haywood via clamav-users
Hi Joel, On Sun, 1 Sep 2019, Joel Esler (jesler) wrote: Alright. I think we’ve beat the proverbial dead horse here. ... I don't think anybody's beating anything here Joel. Just we users, discussing, on the users' list, ways of dealing with an issue. On Sat, 31 Aug 2019, G.W. Haywood

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-01 Thread Joel Esler (jesler) via clamav-users
Alright. I think we’ve beat the proverbial dead horse here. The devs know this is a request and they will get it into their dev queue for examination. Sent from my  iPhone > On Sep 1, 2019, at 13:21, G.W. Haywood via clamav-users > wrote: > > Hi there, > >> On Sun, 1 Sep 2019, Thomas

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-01 Thread G.W. Haywood via clamav-users
Hi there, On Sun, 1 Sep 2019, Thomas Barth via clamav-users wrote: Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: That's exactly what the patch in #10979 does. ... And where can I find this patch? If you navigate to https://bugzilla.clamav.net/show_bug.cgi?id=10979 and

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-01 Thread Matus UHLAR - fantomas
Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: That's exactly what the patch in #10979 does. Unfortunately, although as I've said it's simple enough to apply the patch, it's by no means a simple patch and it would greatly benefit from some serious testing by the community -

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-09-01 Thread Thomas Barth via clamav-users
Am 2019-08-31 20:35, schrieb G.W. Haywood via clamav-users: That's exactly what the patch in #10979 does. Unfortunately, although as I've said it's simple enough to apply the patch, it's by no means a simple patch and it would greatly benefit from some serious testing by the community -

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread G.W. Haywood via clamav-users
Hi there, On Sat, 31 Aug 2019, J.R. via clamav-users wrote: ... I wouldn't call the current design a "bug"... It works as intended. +1 However it would be nice if a fresh DB could be parsed & loaded, then swapped, to prevent service interruption. That's exactly what the patch in #10979

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread J.R. via clamav-users
> Normally postfix gets a response after 3 secondes. > > In the clamav.log I see at the same time, that reloading the database > takes up to two minutes. Yes, reloading the DB can take some time depending on which signature DBs you are using. I can't speak for postfix (I run sendmail), but on my

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Scott Kitterman via clamav-users
On August 31, 2019 4:32:00 PM UTC, Henrik K wrote: >On Sat, Aug 31, 2019 at 12:21:11PM -0400, Scott Kitterman via >clamav-users wrote: >> >> Not to put too fine a point on it, but if you are unhappy with the >service you >> are receiving, you should switch to a different vendor. I suspect

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Henrik K
On Sat, Aug 31, 2019 at 12:21:11PM -0400, Scott Kitterman via clamav-users wrote: > > Not to put too fine a point on it, but if you are unhappy with the service > you > are receiving, you should switch to a different vendor. I suspect it's > unlikely you'll get the same value for money

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread G.W. Haywood via clamav-users
Hi there, On Sat, 31 Aug 2019, Henrik K wrote: On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users wrote: The final responsibility of implementing and testing the issue is still that of the ClamAV team. Agreed. You are really making this much more complex and "scary"

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Scott Kitterman via clamav-users
On Saturday, August 31, 2019 12:04:36 PM EDT Henrik K wrote: > On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users wrote: > > More testing, by people prepared to chip in some effort instead of > > complaining about something that they get for free, would be great. > > The

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Henrik K
On Sat, Aug 31, 2019 at 04:48:54PM +0100, G.W. Haywood via clamav-users wrote: > > More testing, by people prepared to chip in some effort instead of > complaining about something that they get for free, would be great. The final responsibility of implementing and testing the issue is still that

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread G.W. Haywood via clamav-users
Hi there, On Sat, 31 Aug 2019, Henrik K wrote: ... If I encountered a bug like that on some project that I'm maintaining, I would be shamed not to rapidly fix it. If you called it a limitation I could agree, but I guess it's working as designed. I'd call it an issue rather than a fault in

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread G.W. Haywood via clamav-users
Hi there, On Sat, 31 Aug 2019, Henrik K wrote: On Sat, Aug 31, 2019, G.W. Haywood via clamav-users wrote: Well not quite nothing, since you can download the source, apply the patch, and rebuild ClamAV. Sure but it's not reality for majority of users.. While it's good that people try it

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Henrik K
On Sat, Aug 31, 2019 at 11:18:00AM -0400, Michael Orlitzky via clamav-users wrote: > > Micah took the time to answer a question and provide a status update. > It's counterproductive to shame people for being honest. It's perfectly fine to shame a corporation for doing seemingly strange things.

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Michael Orlitzky via clamav-users
On 8/31/19 11:00 AM, Thomas Barth via clamav-users wrote: > > Realy bad attitude of developers! Micah took the time to answer a question and provide a status update. It's counterproductive to shame people for being honest. ___ clamav-users mailing

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Henrik K
On Sat, Aug 31, 2019 at 03:55:30PM +0100, G.W. Haywood via clamav-users wrote: > > Well not quite nothing, since you can download the source, apply the > patch, and rebuild ClamAV. Sure but it's not reality for majority of users.. While it's good that people try it out, I doubt if would take

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Thomas Barth via clamav-users
Am 2019-08-31 16:32, schrieb Henrik K: The reload bug has been known for years, even has a ready patch. Wow, this is a masterpiece ignoring a problem for years :) Thanks for pointing to the bugthread. But nothing you can do about it, ClamAV devs have a mind of their own. Micah Snyder

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread G.W. Haywood via clamav-users
Hi there, On Sat, 31 Aug 2019, Henrik K wrote: The reload bug has been known for years, even has a ready patch. https://bugzilla.clamav.net/show_bug.cgi?id=10979 But nothing you can do about it... Well not quite nothing, since you can download the source, apply the patch, and rebuild

Re: [clamav-users] How to boost clamav? Reloading database results in a talking timeout?

2019-08-31 Thread Henrik K
The reload bug has been known for years, even has a ready patch. https://bugzilla.clamav.net/show_bug.cgi?id=10979 But nothing you can do about it, ClamAV devs have a mind of their own. Atleast servers in your scenario will (hopefully) retry sending. On Sat, Aug 31, 2019 at 04:25:05PM