Re: [clamav-users] Can’t allocate memory error
I am scanning large Data sets for a company. These file systems have hundreds of thousands of files in them. Most files are small in size, <1GB, while a few are large, >10GB. Most files are documents, archives, and executables. I am scanning them to detect if there are any malware. These are virtual machines, running Ubuntu 20.04. The cpu on the esxi host is an Intel Xeon Platinum 828 CPu @2.70GHz. I have in total, 112 logical processors available, and 512 GB of RAM. The message it says is the following: Got command FILDES(7,11) argument RECVTH FILDES command complete THMGR active jobs for ***: 2 THRMGR: Contended, sleeping Nothin under this command, it pauses, then after a couple minutes it will continue, repeating On Tue, Mar 2, 2021 at 9:40 AM G.W. Haywood via clamav-users < clamav-users@lists.clamav.net> wrote: > Hi there, > > On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: > > On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote: > >> On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: > >> > >>> ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate > >>> memory errors” on very large files ( 10GB +). I thought clamdscan > >>> was supposed to skip files that are larger than what you set the > >>> maxfilesize/maxscansize to. > >> > >> Unfortunately this is a known issue: > >> > >> https://bugzilla.clamav.net/show_bug.cgi?id=12374 > >> > >> Have you tried other ways to avoid scanning huge files? > > > > I was not aware of any other way to avoid scanning large files. Where > can I > > find such solutions? > > The operating system offers ways to avoid shooting your own feet. You > could just arrange for all the huge files to be in some corner of the > filesystem which you don't normally scan - which begs the questions > what are you scanning, and why? There will of course be pseudo-files > in your system which you should _never_ scan. The 'find' utility will > let you specify size limits. You will need to spend some quality time > with the 'man' pages to gain familiarity with using standard utilities > in conjunction with something like ClamAV. Using the 'man' pages is > something of an acquired taste, which you do need to acquire if you're > to get the most out of a Linux box. The 'man' page for clamd.conf > contains information about usage of resources. Also there are some > warnings, which to my mind are perhaps a little over the top, but they > serve to remind us that the system's resources may be shared between a > large number of processes; that these processes compete for resources; > and that things can get ugly when there aren't enough to go around. > > The concept of "not scanning a file larger than X bytes" is a bit too > simplistic when talking about scanning with something like ClamAV which > (a) depending on the file type may use different approaches to scanning > and (b) can extract the content from types of file (e.g. Zip, RAR, etc.) > which can contain whole directory structures and also employ compression > techniques, and which as a result are subject to various and sometimes > non-obvious Denial-Of-Service type attacks. So there are numerous clamd > configuration options which permit fine-tuning of the resource usage of > the ClamAV tools. To make the best use of these options you'll need to > be familiar with the your system's resources, and the constraints. > > How much memory does the box have? You'll probably need a gigabyte or > so to store the signature database before you even start a scan, plus > whatever the scanner uses when it scans something - that depends a lot > on what it's scanning. Then if you keep the default configuration to > permit scanning while reloading the databases, another gigabyte will > be used (briefly) every time clamd reloads the database. Note that > the extra memory will not be released until the completion of any scan > which was started before the reload. I'd recommend that if you don't > want to have to work on memory management, four gigabytes of RAM is > about the minimum for a clamd server. The longer it takes to scan a > file, the more likely it is that you'll try to reload the database > during a scan, so if you're short on memory and you want to scan files > which take a long time to scan then it's worth considering the option > to scan data only while a database reload is not taking place. > > -- > > 73, > Ged. > > ___ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml > ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide:
Re: [clamav-users] Can’t allocate memory error
Hi there, On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: On Tue, Mar 2, 2021 at 9:40 AM G.W. Haywood via clamav-users wrote: On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote: On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate memory errors” on very large files ( 10GB +). I thought clamdscan was supposed to skip files that are larger than what you set the maxfilesize/maxscansize to. Unfortunately this is a known issue: https://bugzilla.clamav.net/show_bug.cgi?id=12374 Have you tried other ways to avoid scanning huge files? I was not aware of any other way to avoid scanning large files. Where can I find such solutions? ... You could just arrange for all the huge files to be in some corner of the filesystem which you don't normally scan - which begs the questions what are you scanning, and why? ... My scanners have 16vcpus and 64 GB Ram allocated to them. (Each) I noticed that the clamd process actually began hanging on some of the scanners. Then these are not real harware? If not, one of my first questions would be if I'm getting what some supplier claims that I'm getting. Perhaps it's only "up to" 64GB RAM, and that on a good day. Perhaps there are restrictions on resources that you don't know about. Scans are pretty CPU intensive, if you run many in parallel you might hit a limit fairly easily. ... looking at the log, the only thing it says is “... sleep”. I don't see that exact message anywhere in the ClamAV source. Please may we see the exact log output? A few lines before and after the "sleep" line would probably be useful. After 5-10 minutes it will continue, and then pause again. Do you know how I can troubleshoot this issue? There are many ways to approach such an issue. For example using the command-line and/or configuration file options you could tell the scanner to log more verbosely. You also could use tools like 'top' to observe process activity as it happens - logging the output to files if necessary. If the CPUs are virtual you probably won't need to check their operating temperatures (or have the ability to do so), but I often check things like that on hardware when I give it a trashing. Please allow me to repeat the question what are you scanning, and why? -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Can’t allocate memory error
My scanners have 16vcpus and 64 GB Ram allocated to them. (Each) I noticed that the clamd process actually began hanging on some of the scanners. This slowed the scanning by a lot. looking at the log, the only thing it says is “... sleep”. After 5-10 minutes it will continue, and then pause again. Do you know how I can troubleshoot this issue? On Tue, Mar 2, 2021 at 9:40 AM G.W. Haywood via clamav-users < clamav-users@lists.clamav.net> wrote: > Hi there, > > On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: > > On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote: > >> On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: > >> > >>> ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate > >>> memory errors” on very large files ( 10GB +). I thought clamdscan > >>> was supposed to skip files that are larger than what you set the > >>> maxfilesize/maxscansize to. > >> > >> Unfortunately this is a known issue: > >> > >> https://bugzilla.clamav.net/show_bug.cgi?id=12374 > >> > >> Have you tried other ways to avoid scanning huge files? > > > > I was not aware of any other way to avoid scanning large files. Where > can I > > find such solutions? > > The operating system offers ways to avoid shooting your own feet. You > could just arrange for all the huge files to be in some corner of the > filesystem which you don't normally scan - which begs the questions > what are you scanning, and why? There will of course be pseudo-files > in your system which you should _never_ scan. The 'find' utility will > let you specify size limits. You will need to spend some quality time > with the 'man' pages to gain familiarity with using standard utilities > in conjunction with something like ClamAV. Using the 'man' pages is > something of an acquired taste, which you do need to acquire if you're > to get the most out of a Linux box. The 'man' page for clamd.conf > contains information about usage of resources. Also there are some > warnings, which to my mind are perhaps a little over the top, but they > serve to remind us that the system's resources may be shared between a > large number of processes; that these processes compete for resources; > and that things can get ugly when there aren't enough to go around. > > The concept of "not scanning a file larger than X bytes" is a bit too > simplistic when talking about scanning with something like ClamAV which > (a) depending on the file type may use different approaches to scanning > and (b) can extract the content from types of file (e.g. Zip, RAR, etc.) > which can contain whole directory structures and also employ compression > techniques, and which as a result are subject to various and sometimes > non-obvious Denial-Of-Service type attacks. So there are numerous clamd > configuration options which permit fine-tuning of the resource usage of > the ClamAV tools. To make the best use of these options you'll need to > be familiar with the your system's resources, and the constraints. > > How much memory does the box have? You'll probably need a gigabyte or > so to store the signature database before you even start a scan, plus > whatever the scanner uses when it scans something - that depends a lot > on what it's scanning. Then if you keep the default configuration to > permit scanning while reloading the databases, another gigabyte will > be used (briefly) every time clamd reloads the database. Note that > the extra memory will not be released until the completion of any scan > which was started before the reload. I'd recommend that if you don't > want to have to work on memory management, four gigabytes of RAM is > about the minimum for a clamd server. The longer it takes to scan a > file, the more likely it is that you'll try to reload the database > during a scan, so if you're short on memory and you want to scan files > which take a long time to scan then it's worth considering the option > to scan data only while a database reload is not taking place. > > -- > > 73, > Ged. > > ___ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml > ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Can’t allocate memory error
Hi there, On Tue, 2 Mar 2021, Michael Kyriacou via clamav-users wrote: On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users wrote: On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate memory errors” on very large files ( 10GB +). I thought clamdscan was supposed to skip files that are larger than what you set the maxfilesize/maxscansize to. Unfortunately this is a known issue: https://bugzilla.clamav.net/show_bug.cgi?id=12374 Have you tried other ways to avoid scanning huge files? I was not aware of any other way to avoid scanning large files. Where can I find such solutions? The operating system offers ways to avoid shooting your own feet. You could just arrange for all the huge files to be in some corner of the filesystem which you don't normally scan - which begs the questions what are you scanning, and why? There will of course be pseudo-files in your system which you should _never_ scan. The 'find' utility will let you specify size limits. You will need to spend some quality time with the 'man' pages to gain familiarity with using standard utilities in conjunction with something like ClamAV. Using the 'man' pages is something of an acquired taste, which you do need to acquire if you're to get the most out of a Linux box. The 'man' page for clamd.conf contains information about usage of resources. Also there are some warnings, which to my mind are perhaps a little over the top, but they serve to remind us that the system's resources may be shared between a large number of processes; that these processes compete for resources; and that things can get ugly when there aren't enough to go around. The concept of "not scanning a file larger than X bytes" is a bit too simplistic when talking about scanning with something like ClamAV which (a) depending on the file type may use different approaches to scanning and (b) can extract the content from types of file (e.g. Zip, RAR, etc.) which can contain whole directory structures and also employ compression techniques, and which as a result are subject to various and sometimes non-obvious Denial-Of-Service type attacks. So there are numerous clamd configuration options which permit fine-tuning of the resource usage of the ClamAV tools. To make the best use of these options you'll need to be familiar with the your system's resources, and the constraints. How much memory does the box have? You'll probably need a gigabyte or so to store the signature database before you even start a scan, plus whatever the scanner uses when it scans something - that depends a lot on what it's scanning. Then if you keep the default configuration to permit scanning while reloading the databases, another gigabyte will be used (briefly) every time clamd reloads the database. Note that the extra memory will not be released until the completion of any scan which was started before the reload. I'd recommend that if you don't want to have to work on memory management, four gigabytes of RAM is about the minimum for a clamd server. The longer it takes to scan a file, the more likely it is that you'll try to reload the database during a scan, so if you're short on memory and you want to scan files which take a long time to scan then it's worth considering the option to scan data only while a database reload is not taking place. -- 73, Ged. ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Can’t allocate memory error
Hello Michael
Le 02/03/2021 à 13:44, Michael Kyriacou via clamav-users a écrit :
I was not aware of any other way to avoid scanning large files. Where
can I find such solutions?
As an example scan all files below 50Mb :
find /your_path -type f -size -50M|parallel clamdscan -mi --fdpass
--no-summary {}
--
Cordialement / Best regards,
Arnaud Jacques
Gérant de SecuriteInfo.com
Téléphone : +33-(0)3.60.47.09.81
E-mail : a...@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom
Signatures for ClamAV antivirus : http://ow.ly/LqfdL
___
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
Re: [clamav-users] Can’t allocate memory error
I was not aware of any other way to avoid scanning large files. Where can I find such solutions? On Tue, Mar 2, 2021 at 4:08 AM G.W. Haywood via clamav-users < clamav-users@lists.clamav.net> wrote: > Hi there, > > On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: > > > ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate > > memory errors” on very large files ( 10GB +). I thought clamdscan > > was supposed to skip files that are larger than what you set the > > maxfilesize/maxscansize to. > > Unfortunately this is a known issue: > > https://bugzilla.clamav.net/show_bug.cgi?id=12374 > > Have you tried other ways to avoid scanning huge files? > > -- > > 73, > Ged. > OA > > ___ > > clamav-users mailing list > clamav-users@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-users > > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml > ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [clamav-users] Can’t allocate memory error
Hi there, On Mon, 1 Mar 2021, Michael Kyriacou via clamav-users wrote: ... clamav 103.1 on ubuntu 20.04. I am getting “can’t allocate memory errors” on very large files ( 10GB +). I thought clamdscan was supposed to skip files that are larger than what you set the maxfilesize/maxscansize to. Unfortunately this is a known issue: https://bugzilla.clamav.net/show_bug.cgi?id=12374 Have you tried other ways to avoid scanning huge files? -- 73, Ged. OA ___ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
