Re: [Clamav-users] Can I give clam a list of files to scan
On 9/8/06, tBB [EMAIL PROTECTED] wrote: The problem is that clamscan wants the files or directories passed to it via the command line, not via stdin-- besides which, Windows has a fairly limited max length for the command line. Actually it's not that limited (but still too limited for this purpose I guess). Windows XP/2k3 has a max command line length of 8191 characters and Win2k/NT has a limit of 2047. However, if the list of files can be provided externally it's not much of a problem. A simple .bat should do: @echo off for /F %%a in (files_to_scan.list) do clamdscan.exe %%a This example works for one filename per line. If you rather prefer to separate the filenames by some character, let's say ; it would look like: for /F delims=; %%a in (files_to_scan.list) do clamdscan.exe %%a Best regards, Nico So ultimately clamscan.exe will not take stdin. I will just have to run it like you state above... multiple time! By reloading clamscan.exe in and out of memory for every file at which point am I better off just scanning the entire drive recursively? Thanks again for every ones insight! -- +--+ Q: Because it reverses the logical flow of conversation. A: Why is putting a reply at the top of the message frowned upon? ___ http://lurker.clamav.net/list/clamav-users.html -- ** *** * *** *** ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Sep 11, 2006, at 2:50 PM, Matthew Dettinger wrote: So ultimately clamscan.exe will not take stdin. I will just have to run it like you state above... multiple time! By reloading clamscan.exe in and out of memory for every file at which point am I better off just scanning the entire drive recursively? Well, clamscan will read stuff from stdin and scan it, but only as a single file. That is, clamscan somefile ...and: clamscan somefile ...do the same thing, or nearly. Clamscan won't read stuff from stdin as if it were a list of filenames to be scanned, although perhaps an option to provide that capability would be useful. -- -Chuck ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
Matthew Dettinger wrote: So ultimately clamscan.exe will not take stdin. I will just have to run it like you state above... multiple time! By reloading clamscan.exe in and out of memory for every file at which point am I better off just scanning the entire drive recursively? Yes, Clam(D)Scan won't take more than one file from stdin. If you can't run the ClamD daemon and scan the files by using ClamDscan for some reason it would eventually be better to scan the entire drive, depending on the number of files to scan as was pointed out previously. Best regards, Nico -- +--+ Q: Because it reverses the logical flow of conversation. A: Why is putting a reply at the top of the message frowned upon? ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
Matthew Dettinger wrote: On 9/8/06, tBB [EMAIL PROTECTED] wrote: The problem is that clamscan wants the files or directories passed to it via the command line, not via stdin-- besides which, Windows has a fairly limited max length for the command line. Actually it's not that limited (but still too limited for this purpose I guess). Windows XP/2k3 has a max command line length of 8191 characters and Win2k/NT has a limit of 2047. However, if the list of files can be provided externally it's not much of a problem. A simple .bat should do: @echo off for /F %%a in (files_to_scan.list) do clamdscan.exe %%a This example works for one filename per line. If you rather prefer to separate the filenames by some character, let's say ; it would look like: for /F delims=; %%a in (files_to_scan.list) do clamdscan.exe %%a Best regards, Nico So ultimately clamscan.exe will not take stdin. I will just have to run it like you state above... multiple time! By reloading clamscan.exe in and out of memory for every file at which point am I better off just scanning the entire drive recursively? Thanks again for every ones insight! A simple Perl script that makes a socket connection to Clamd can be written that is run once per scan. It can read a pre-made list of files to scan or it can crawl the hard disk (very inefficient) to build a list. The Perl script can then walk the array of files and submit them one at a time to Clamd. Clamd has to run as administrator so that it can open any files given to it. This is also true for Python, PHP, and even VB. You just need a socket-capable language. This avoids reloading clamscan thousands of times, or scanning file system areas that don't require scanning. dp ___ http://lurker.clamav.net/list/clamav-users.html
[Clamav-users] Can I give clam a list of files to scan
Basically I want to continue on with what some of the people on this list were saying about using tripwire (or something similar) to just scan files on a system that have changed. I wouldn't want to call clamscan multiple times each with one file, but rather call it once with a big list of files to scan. mcd -- ** *** * *** *** ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] Can I give clam a list of files to scan
mcd wrote: Basically I want to continue on with what some of the people on this list were saying about using tripwire (or something similar) to just scan files on a system that have changed. I wouldn't want to call clamscan multiple times each with one file, but rather call it once with a big list of files to scan. I don't think so. What you can do is start clamd and then call clamdscan for each file. -- Bowie ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] Can I give clam a list of files to scan
find (options) | xargs clamscan So to search all files in home find /home/ |xargs clamscan Or to scan only certain files of specified size (thanks to Noel Jones for this one) find / -type f -size N | xargs clamscan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of mcd Sent: 08 September 2006 04:22 PM To: ClamAV users ML Subject: [Clamav-users] Can I give clam a list of files to scan Basically I want to continue on with what some of the people on this list were saying about using tripwire (or something similar) to just scan files on a system that have changed. I wouldn't want to call clamscan multiple times each with one file, but rather call it once with a big list of files to scan. mcd -- ** *** * *** *** ___ http://lurker.clamav.net/list/clamav-users.html ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
mcd wrote: Basically I want to continue on with what some of the people on this list were saying about using tripwire (or something similar) to just scan files on a system that have changed. I wouldn't want to call clamscan multiple times each with one file, but rather call it once with a big list of files to scan. Just feed the file names to clamscan. clamscan file1 file2 file3 etc Or if the files are listed in some file (file_list for example) somewhere you can always do: clamscan `cat file_list` Steve ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] Can I give clam a list of files to scan
Or if you maintain a file with the filenames and paths that you want to scan, you can use cat to output each line of that file to clamscan in the same fashion. cat filename | xargs clamscan ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
mcd wrote: Basically I want to continue on with what some of the people on this list were saying about using tripwire (or something similar) to just scan files on a system that have changed. I wouldn't want to call clamscan multiple times each with one file, but rather call it once with a big list of files to scan. mcd clamscan *.tar *.zip *.exe works fine, for example. You can also provide specific directories to scan recursively. I think the only limiting factor is the length of your command line allowed by the OS.dp clamscan --help describes the use of --include and --exclude options to further prune or extend the pattern. dp ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Fri, 8 Sep 2006, mcd wrote: Basically I want to continue on with what some of the people on this list were saying about using tripwire (or something similar) to just scan files on a system that have changed. I wouldn't want to call clamscan multiple times each with one file, but rather call it once with a big list of files to scan. mcd -- ** *** * *** *** ___ http://lurker.clamav.net/list/clamav-users.html run freshclam to ensure the virus definition file are up to date. cd to the directory of files you want scaned. clamscan -ir --remove ./ note: the --remove option will rm the infected files from the directory. There are no old CARELESS pilots or electricians. ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
This is an interesting approach, but let me explain a little more. I will be running md5sums for every file on a system. I will then compare that list of md5sums against a list of md5sums that are know to be virus free. The files that do not have valid md5sums in the database will then need to be scanned. In a lot of situations this list of files will be in the thousands. At this point would I be better off calling clamscan to scan the entire disk, or call clamscan 10,000+ times with unknown files? I hope I am making this clear. Thanks for all of your help. mcd On 9/8/06, Barry Gill [EMAIL PROTECTED] wrote: Or if you maintain a file with the filenames and paths that you want to scan, you can use cat to output each line of that file to clamscan in the same fashion. cat filename | xargs clamscan ___ http://lurker.clamav.net/list/clamav-users.html -- ** *** * *** *** ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
mcd wrote: This is an interesting approach, but let me explain a little more. I will be running md5sums for every file on a system. I will then compare that list of md5sums against a list of md5sums that are know to be virus free. The files that do not have valid md5sums in the database will then need to be scanned. In a lot of situations this list of files will be in the thousands. At this point would I be better off calling clamscan to scan the entire disk, or call clamscan 10,000+ times with unknown files? I hope I am making this clear. Thanks for all of your help. I do this with TripWire. TripWire does checksum and more to determine if a file has changed, and provides a comprehensive report you can use with ClamAV. You can also start an instance of clamd that runs as user root and feed it with clamdscan. It would be prudent to kill that instance at the end of the scan. This loads the pattern file once and reuses it. This introduces other issues. See clamdscan --help. dp ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] Can I give clam a list of files to scan
mcd wrote: On 9/8/06, Barry Gill [EMAIL PROTECTED] wrote: cat filename | xargs clamscan This is an interesting approach, but let me explain a little more. I will be running md5sums for every file on a system. I will then compare that list of md5sums against a list of md5sums that are know to be virus free. The files that do not have valid md5sums in the database will then need to be scanned. In a lot of situations this list of files will be in the thousands. At this point would I be better off calling clamscan to scan the entire disk, or call clamscan 10,000+ times with unknown files? I hope I am making this clear. Thanks for all of your help. Calling clamscan 10,000 times is a very bad idea. Considering that it takes clamscan 1.4 seconds to scan a single small file on my system... Instead, you should use the clamd daemon and then call clamdscan 10,000 times. It only takes clamdscan 0.005 seconds to scan that same single file. You are probably better off scanning the list of files unless that list represents 90% or more of your drive. There is very little overhead to calling clamdscan. -- Bowie ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Fri, September 8, 2006 10:42 am, mcd said: This is an interesting approach, but let me explain a little more. I will be running md5sums for every file on a system. I will then compare that list of md5sums against a list of md5sums that are know to be virus free. The files that do not have valid md5sums in the database will then need to be scanned. In a lot of situations this list of files will be in the thousands. At this point would I be better off calling clamscan to scan the entire disk, or call clamscan 10,000+ times with unknown files? I hope I am making this clear. Thanks for all of your help. (Please don't top-post, it makes it hard to usefully trim the discussion.) Running clamscan over the entire disk is still overkill in this situation; even if 10% if only the disk does *not* need scanning, I'd restrict to just the other 90%. The response you replied to is probably what I'd use for this scenario: Either write a file with the paths of the files to be scanned and pipe that to clamscan, or just pipe it directly from you comparison script. That's only calling clamscan once, so you aren't hitting the startup costs over and over. (Which *would* be a problem.) If doing one of those is not possible, and you must call a program seperately for each file, use clamdscan to do the scanning, but be aware of permissions issues based on who started clamd. Daniel T. Staal --- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. --- ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
At 09:42 AM 9/8/2006, mcd wrote: This is an interesting approach, but let me explain a little more. I will be running md5sums for every file on a system. I will then compare that list of md5sums against a list of md5sums that are know to be virus free. The files that do not have valid md5sums in the database will then need to be scanned. In a lot of situations this list of files will be in the thousands. At this point would I be better off calling clamscan to scan the entire disk, or call clamscan 10,000+ times with unknown files? I hope I am making this clear. Thanks for all of your help. cat big.list.of.files | xargs clamscan man xargs to see a description and options. You could alternately use clamdscan, but then you may run into permission problems as clamd usually isn't run as root. -- Noel Jones ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
Noel Jones wrote: At 09:42 AM 9/8/2006, mcd wrote: This is an interesting approach, but let me explain a little more. I will be running md5sums for every file on a system. I will then compare that list of md5sums against a list of md5sums that are know to be virus free. The files that do not have valid md5sums in the database will then need to be scanned. In a lot of situations this list of files will be in the thousands. At this point would I be better off calling clamscan to scan the entire disk, or call clamscan 10,000+ times with unknown files? I hope I am making this clear. Thanks for all of your help. cat big.list.of.files | xargs clamscan xargs is still limited by max line length, so this needs to be done with care. Perl can also be used in place of clamdscan to feed file names to clamd (which must be run as root). The advantage of Perl is it can iterate over an array and of course manage all the logging. dp dp ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
At 10:11 AM 9/8/2006, Dennis Peterson wrote: Noel Jones wrote: cat big.list.of.files | xargs clamscan xargs is still limited by max line length, so this needs to be done with care. Perl can also be used in place of clamdscan to feed file names to clamd (which must be run as root). The advantage of Perl is it can iterate over an array and of course manage all the logging. The general purpose of xargs is to split a huge list into chunks less than max line length - lather, rinse, and repeat. While the xargs command itself (clamscan in the example) is subject to max line length, the input (big.list.of.files) is not. -- Noel Jones ___ http://lurker.clamav.net/list/clamav-users.html
RE: [Clamav-users] Can I give clam a list of files to scan
xargs is still limited by max line length, so this needs to be done with care. Perl can also be used in place of clamdscan to feed file names to clamd (which must be run as root). The advantage of Perl is it can iterate over an array and of course manage all the logging. Also, depending on how much resource you have available, you can runa script to call the md5 checksum, verify it and if it has changed pass it thorugh to clamscan without using xargs at all. A simple find / |clamscan will scan every file in the filesystem, but will also chew through resource unnecessarily. So, depending on how often, how many changes, it may be well worth your while to set up a decent script that runs clamscan just after calculating your md5 sums and validating them against your known good db, and takes any file that reports OK from clamscan and adding its name and md5 sum to the DB. Realistically, unless your full system is going to be changing all the time, this process should take less and less time as your md5 DB covers more and more of the files ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
At 10:11 AM 9/8/2006, Dennis Peterson wrote: Noel Jones wrote: cat big.list.of.files | xargs clamscan xargs is still limited by max line length, so this needs to be done with care. Perl can also be used in place of clamdscan to feed file names to clamd (which must be run as root). The advantage of Perl is it can iterate over an array and of course manage all the logging. The general purpose of xargs is to split a huge list into chunks less than max line length - lather, rinse, and repeat. While the xargs command itself (clamscan in the example) is subject to max line length, the input (big.list.of.files) is not. I brain-farted - the output is piped to xargs so the command line length is not a factor. Hopefully the list is also properly escaped and or quoted. xargs is pretty unhappy with filenames that have special characters in them, or spaces. This is true no matter how the list is submitted to the scanner. This is the gripe I have against using 'find' without sanitizing the output. dp ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Fri, Sep 08, 2006 at 10:05:48AM -0700, Dennis Peterson wrote: Hopefully the list is also properly escaped and or quoted. xargs is pretty unhappy with filenames that have special characters in them, or spaces. This is true no matter how the list is submitted to the scanner. This is the gripe I have against using 'find' without sanitizing the output. Thats why we have: find -print0 | xargs -0 hk ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Fri, Sep 08, 2006 at 10:05:48AM -0700, Dennis Peterson wrote: Hopefully the list is also properly escaped and or quoted. xargs is pretty unhappy with filenames that have special characters in them, or spaces. This is true no matter how the list is submitted to the scanner. This is the gripe I have against using 'find' without sanitizing the output. Thats why we have: find -print0 | xargs -0 Assumes Linux? dp ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Sep 8, 2006, at 11:34 AM, Dennis Peterson wrote: On Fri, Sep 08, 2006 at 10:05:48AM -0700, Dennis Peterson wrote: Hopefully the list is also properly escaped and or quoted. xargs is pretty unhappy with filenames that have special characters in them, or spaces. This is true no matter how the list is submitted to the scanner. This is the gripe I have against using 'find' without sanitizing the output. Thats why we have: find -print0 | xargs -0 Assumes Linux? or Unix or OS X dp ___ http://lurker.clamav.net/list/clamav-users.html ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On 9/8/06, Philip Ershler [EMAIL PROTECTED] wrote: On Sep 8, 2006, at 11:34 AM, Dennis Peterson wrote: On Fri, Sep 08, 2006 at 10:05:48AM -0700, Dennis Peterson wrote: Hopefully the list is also properly escaped and or quoted. xargs is pretty unhappy with filenames that have special characters in them, or spaces. This is true no matter how the list is submitted to the scanner. This is the gripe I have against using 'find' without sanitizing the output. Thats why we have: find -print0 | xargs -0 Assumes Linux? or Unix or OS X Sorry this is on a windows system. Can I just cat the file list and pipe it to the windows version of clamscan? I believe it takes stdin? -- ** *** * *** *** ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Sep 8, 2006, at 10:34 AM, Dennis Peterson wrote: Henrik Krohns [EMAIL PROTECTED] wrote: Thats why we have: find -print0 | xargs -0 Assumes Linux? No. This -print0 option first appeared in GNU find before Linux existed, as far as I can tell. It was adopted into BSD versions of find around 1993 to 1995: http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/find/option.c#rev1.4 http://www.freebsd.org/cgi/cvsweb.cgi/src/usr.bin/find/option.c#rev1.2 -- -Chuck ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Sep 8, 2006, at 11:25 AM, mcd wrote: Sorry this is on a windows system. Can I just cat the file list and pipe it to the windows version of clamscan? I believe it takes stdin? The problem is that clamscan wants the files or directories passed to it via the command line, not via stdin-- besides which, Windows has a fairly limited max length for the command line. However, if you would like to obtain a working version of find and xargs for Windows, consider installing Cygwin from: http://www.cygwin.com This will probably make working with Unix-derived software like ClamAV easier... -- -Chuck ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
- Original Message - From: Chuck Swiger [EMAIL PROTECTED] On Sep 8, 2006, at 11:25 AM, mcd wrote: Sorry this is on a windows system. Can I just cat the file list and pipe it to the windows version of clamscan? I believe it takes stdin? The problem is that clamscan wants the files or directories passed to it via the command line, not via stdin-- besides which, Windows has a fairly limited max length for the command line. However, if you would like to obtain a working version of find and xargs for Windows, consider installing Cygwin from: http://www.cygwin.com This will probably make working with Unix-derived software like ClamAV easier... Or maybe even better yet, instead of installing cygwin, use the native UNIX tools for Win32 from http://unxutils.sourceforge.net/, which includes a native build of xargs and most other UNIX utilities. Bill ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Sep 8, 2006, at 10:34 AM, Dennis Peterson wrote: Henrik Krohns [EMAIL PROTECTED] wrote: Thats why we have: find -print0 | xargs -0 Assumes Linux? No. This -print0 option first appeared in GNU find before Linux existed, as far as I can tell. It was adopted into BSD versions of find around 1993 to 1995: Seems not to work in Solaris. dp ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Fri, 8 Sep 2006, Dennis Peterson wrote: Seems not to work in Solaris. As a Solaris fan -- you REALLY want to install gnu find, and grep, and fileutils. At least. Sun still for whatever reason doesn't support many newer options, newer being post 1989. == Chris Candreva -- [EMAIL PROTECTED] -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/ ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Fri, 8 Sep 2006, Dennis Peterson wrote: Seems not to work in Solaris. As a Solaris fan -- you REALLY want to install gnu find, and grep, and fileutils. At least. Sun still for whatever reason doesn't support many newer options, newer being post 1989. You can't imagine the Change Management hell we'd have to go through to do this to all our production and dev/test environments. It really isn't worth the aggravation. And so far it hasn't been necessary as the workarounds have been in place for years. But for the OP's application I still prefer Perl and it is in Solaris, and can use direct socket communication with clamd. (to drag this back on topic :) dp ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
On Sep 8, 2006, at 12:19 PM, Dennis Peterson wrote: No. This -print0 option first appeared in GNU find before Linux existed, as far as I can tell. It was adopted into BSD versions of find around 1993 to 1995: Seems not to work in Solaris. Agreed-- Solaris and AIX are the two platforms I know of and still use which have not updated many of the BSD-derived tools since 1990 or so. Many Solaris or AIX users install GNU find fileutils as a result -- -Chuck ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Can I give clam a list of files to scan
The problem is that clamscan wants the files or directories passed to it via the command line, not via stdin-- besides which, Windows has a fairly limited max length for the command line. Actually it's not that limited (but still too limited for this purpose I guess). Windows XP/2k3 has a max command line length of 8191 characters and Win2k/NT has a limit of 2047. However, if the list of files can be provided externally it's not much of a problem. A simple .bat should do: @echo off for /F %%a in (files_to_scan.list) do clamdscan.exe %%a This example works for one filename per line. If you rather prefer to separate the filenames by some character, let's say ; it would look like: for /F delims=; %%a in (files_to_scan.list) do clamdscan.exe %%a Best regards, Nico -- +--+ Q: Because it reverses the logical flow of conversation. A: Why is putting a reply at the top of the message frowned upon? ___ http://lurker.clamav.net/list/clamav-users.html
