Re: [clamav-users] Radically Different Scan Times

2019-04-05 Thread Al Varnell via clamav-users 
Addressed earlier today:

>

-Al-

> On Apr 5, 2019, at 20:18, Michael Newman via clamav-users 
>  wrote:
> 
> MacOS 10.14.4 - 2017 iMac
> ClamAV 0.101.1 (Updated today: ClamAV 0.101.2/25410/Fri Apr  5 14:58:26 2019)
> 
> Yesterday’s results:
> 
> --- SCAN SUMMARY ---
> Known viruses: 6101439
> Engine version: 0.101.1
> Scanned directories: 227591
> Scanned files: 594694
> Infected files: 1
> Total errors: 35
> Data scanned: 63016.47 MB
> Data read: 92969.95 MB (ratio 0.68:1)
> Time: 12755.457 sec (212 m 35 s)
> 
> Today’s results:
> 
> --- SCAN SUMMARY ---
> Known viruses: 6110476
> Engine version: 0.101.1
> Scanned directories: 227492
> Scanned files: 592573
> Infected files: 1
> Total errors: 35
> Data scanned: 63134.07 MB
> Data read: 93149.45 MB (ratio 0.68:1)
> Time: 36218.816 sec (603 m 38 s)
> 
> (Note that the "infected file" is the Eicar-Test-Signature.)
> 
> Even though the number of files and amount of data scanned is about the same, 
> the scan took almost three times as long. I’ve never seen this before. 
> Normally the scan results are there when I wake up in the morning. But, not 
> today.
> 
> I have no idea what to look for here. I’ve not changed anything about the 
> scan nor were other IO intensive jobs running overnight.
> 
> Any suggestions?


smime.p7s
Description: S/MIME cryptographic signature

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] Radically Different Scan Times

2019-04-05 Thread Michael Newman via clamav-users 
MacOS 10.14.4 - 2017 iMac
ClamAV 0.101.1 (Updated today: ClamAV 0.101.2/25410/Fri Apr  5 14:58:26 2019)

Yesterday’s results:

--- SCAN SUMMARY ---
Known viruses: 6101439
Engine version: 0.101.1
Scanned directories: 227591
Scanned files: 594694
Infected files: 1
Total errors: 35
Data scanned: 63016.47 MB
Data read: 92969.95 MB (ratio 0.68:1)
Time: 12755.457 sec (212 m 35 s)

Today’s results:

--- SCAN SUMMARY ---
Known viruses: 6110476
Engine version: 0.101.1
Scanned directories: 227492
Scanned files: 592573
Infected files: 1
Total errors: 35
Data scanned: 63134.07 MB
Data read: 93149.45 MB (ratio 0.68:1)
Time: 36218.816 sec (603 m 38 s)

(Note that the "infected file" is the Eicar-Test-Signature.)

Even though the number of files and amount of data scanned is about the same, 
the scan took almost three times as long. I’ve never seen this before. Normally 
the scan results are there when I wake up in the morning. But, not today.

I have no idea what to look for here. I’ve not changed anything about the scan 
nor were other IO intensive jobs running overnight.

Any suggestions?



___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Malformed pattern daily.ldb version 25410

2019-04-05 Thread David Shrimpton via clamav-users 
This appears to be a different problem than the sigtool --list problem on daily

I think it may  be a problem with integrity of downloaded file and not an 
incompatibility
of that file with clamav version or something wrong with a sig in the file.
Testing the main.cvd file may be good first step.

It appears to be reported that  the  main.cvd downloaded  is corrupted:

>> Fri Apr  5 14:17:59 2019 -> *Trying to download
>>http://db.US.clamav.net/main.cvd (IP: 104.16.219.84)
>> Fri Apr  5 14:18:12 2019 -> Downloading main.cvd [100%]
>> Fri Apr  5 14:18:12 2019 -> ^[LibClamAV] cli_cvdload: Corrupted CVD header
>>Fri Apr  5 14:18:12 2019 -> !Verification: Malformed database

Some things that may help debug:

# download the main.cvd manually eg if have unix wget or curl

wget http://db.US.clamav.net/main.cvd

# check the size , is it zero length or improbably small ? Did wget report 
errors.

# Test main.cvd with sigtool look for errors or sensible output as below.

sigtool --info main.cvd

File: main.cvd
Build time: 07 Jun 2017 17:38 -0400
Version: 58
Signatures: 4566249
Functionality level: 60
Builder: sigmgr
MD5: 57462fd73f1cfdb356b9dca66da2b732
Digital signature: 
KWRdhTG+Own6ohh0wn5+vqg1d8ULKCxxxQeKuSA155B3ijxBKgf+bV3IXPcmZrIBUDn1xi8FmyvB63UieykwN/Avq5mTjHIVO8zFnC7wVF7dhdcEYn9Nt+Pmk/HXXx0voylYkidvgZmrxI8jx4a/Re6n3hHQJoCZrkHM15GER8j
Verification OK.

# examine main.cvd with binary editor eg xxd
main.cvd should have a 512 byte header then a gzipped tar file containing the 
database files and a main.info
The header has : separated fields .  About the 4th field should look like an 
md5sum like above 57462fd73f1cfdb356b9dca66da2b732.
This is the md5sum of the gz that follows the header.   The header seems to end 
with space padding.
about the 5th field should look like a the value of Digital signature: above.   
You should see the Builder field eg sigmgr above.

I think sigtool has verified the signature above.  If file has been altered 
then verification failure might be reported. eg
is db.US.clamav.net the real clamav mirror site or an imposter.WARNING if 
the file isn't verifying it may be malicious
eg a compression bomb , a malicious archive , an exploit against some of the 
tools below and it might be dangerous to run
some of the tools below against it.  Remember only http was used not https to 
get the file so site might be bogus and file
could be anything.

# Extract gz from main.cvd eg with ddand calling the gz main.gz
ie strip off the 512 byte header at start

dd if=main.cvd of=main.gz skip=1 bs=512

# test the gz
gunzip -t main.gz

# extract gz (it will be large eg 3 times size of the gz on my example)
gunzip main.gz  

# this should give a tar file called main for my example
# test the tar file  (my tar reports improbable dates)
tar tvf main
-- 0/0   17992 1970-01-01 10:00 COPYING
-- 0/01060 1970-01-01 10:00 main.info
-- 0/0 3649543 1970-01-01 10:00 main.hdb
-- 0/024806499 1970-01-01 10:00 main.hsb
etc

#  try extracting main.info and some of the database files 
tar xf main main.info

# main.info contains sha256sum for each database file.
# test the extracted database files have same 256 sum
eg from main.info

main.sfp:87:ded8b3b340e2da8415f1409959abb54725afad137a66e938080c7c95a9413128

sha256sum main.sfp
ded8b3b340e2da8415f1409959abb54725afad137a66e938080c7c95a9413128  main.sfp

If a sha256 doesn't  match  that database file is corrupted or altered or 
main.sfp is wrong

You could look at a database file eg main.ndb with text editor or xxd  ans 
should see lines looking
like clamav signatures.Try 'file main.ndb'  first to make sure is a text 
file . Corrupted file might be binary
and trash your terminal or editor.

If the main.cvd appears to be OK then maybe the problem is it isn't compatible 
with clamav version.
You'd need to look at things like version and functionality level from the 
sigtool output and decide
if this is what is expected for a current main.cvd.If it is then I guess 
that incompatible main.cvd
or some faulty sig in main.cvd might be the issue. 


>> Is there a way to go back to daily-25409, for example, other than using
backups?  I looked at the FAQ,

If the main.cvd is corrupted I doubt freshclam would replace existing database 
files and
sigtool --versionmay show you are already on daily-25409  or earlier.

Note if running

freshclam --datadir

I think any settings other than database location from freshclam.conf would 
apply.   So if you were just trying to
get an example main.cvd you might see side effects you don't want  like 
freshclam writing to a configured log file
or trying to HUP your clamd or writing a mirrors.dat 

David Shrimpton


___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

Re: [clamav-users] Clamav for educational institutions ?

2019-04-05 Thread Joel Esler (jesler) via clamav-users 
Sorry if I implied otherwise.  I meant Fedora and their difficulties with 
unrar.  I am a big supporter of your Scott, this you know. 

Sent from my  iPad

> On Apr 5, 2019, at 20:53, Scott Kitterman via clamav-users 
>  wrote:
> 
> On a Debian system with non-free enabled, it only takes "apt install 
> libclamunrar9" to get the full unrar capability.  It's still a better 
> solution for Debian users to use the packaged version.
> 
> There are a few exceptions (for example, getting 0.101 and libclamav9 
> transitioned into our stable release is taking some time, due to reverse 
> depends and patching needed for the changed API, so if one really needs that 
> now, then by all means build from source), but generally Debian users are 
> better served by the O/S integration provided through the packaging system.
> 
> We have an exception to the usual rule about no new versions of packages in 
> stable releases for clamav, so the usual reason, not wanting to be stuck with 
> an old version of the package doesn't generally apply.
> 
> I don't want to get into an extended argument about which is better, but I 
> think Debian does a pretty good job as a clamav distributor.
> 
> Scott K
> 
>> On April 6, 2019 12:21:05 AM UTC, "Joel Esler (jesler)"  
>> wrote:
>> Correct.  Which is why we recommend people compile from source for full
>> functionality.  
>> 
>> Sent from my  iPhone
>> 
>>> On Apr 5, 2019, at 20:12, Scott Kitterman via clamav-users
>>  wrote:
>>> 
>>> The unrar stuff is still free to use.
>>> 
>>> Due to modification restrictions Debian splits it off into the
>> unofficial non-free repository.
>>> 
>>> Scott K
>>> 
 On April 6, 2019 12:03:03 AM UTC, "J.R. via clamav-users"
>>  wrote:
 I just doubled checked, but I don't see a LICENSE file in the
 clamav-0.101.2.tar.gz archive???
 
 EDIT - There is the GPLv2 contained in the COPYING file. I just
 realized each of those files gives the licence for each part of
 ClamAV. Probably the most notable is the unrar licence, which if I
 recall RHEL/CentOS disables due to licence conflicts?
 
 
 
 On Fri, Apr 5, 2019 at 4:30 PM Joel Esler (jesler)
>> 
 wrote:
> 
> That’s the content on the website.  ClamAV, the software, is
>> governed
 by the GPLv2 and other associates licenses as indicated by the
>> LICENSE
 file contained therein.
> 
> ___
> 
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml


smime.p7s
Description: S/MIME cryptographic signature

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Clamav for educational institutions ?

2019-04-05 Thread Scott Kitterman via clamav-users 
On a Debian system with non-free enabled, it only takes "apt install 
libclamunrar9" to get the full unrar capability.  It's still a better solution 
for Debian users to use the packaged version.

There are a few exceptions (for example, getting 0.101 and libclamav9 
transitioned into our stable release is taking some time, due to reverse 
depends and patching needed for the changed API, so if one really needs that 
now, then by all means build from source), but generally Debian users are 
better served by the O/S integration provided through the packaging system.

We have an exception to the usual rule about no new versions of packages in 
stable releases for clamav, so the usual reason, not wanting to be stuck with 
an old version of the package doesn't generally apply.

I don't want to get into an extended argument about which is better, but I 
think Debian does a pretty good job as a clamav distributor.

Scott K

On April 6, 2019 12:21:05 AM UTC, "Joel Esler (jesler)"  
wrote:
>Correct.  Which is why we recommend people compile from source for full
>functionality.  
>
>Sent from my  iPhone
>
>> On Apr 5, 2019, at 20:12, Scott Kitterman via clamav-users
> wrote:
>> 
>> The unrar stuff is still free to use.
>> 
>> Due to modification restrictions Debian splits it off into the
>unofficial non-free repository.
>> 
>> Scott K
>> 
>>> On April 6, 2019 12:03:03 AM UTC, "J.R. via clamav-users"
> wrote:
>>> I just doubled checked, but I don't see a LICENSE file in the
>>> clamav-0.101.2.tar.gz archive???
>>> 
>>> EDIT - There is the GPLv2 contained in the COPYING file. I just
>>> realized each of those files gives the licence for each part of
>>> ClamAV. Probably the most notable is the unrar licence, which if I
>>> recall RHEL/CentOS disables due to licence conflicts?
>>> 
>>> 
>>> 
>>> On Fri, Apr 5, 2019 at 4:30 PM Joel Esler (jesler)
>
>>> wrote:
 
 That’s the content on the website.  ClamAV, the software, is
>governed
>>> by the GPLv2 and other associates licenses as indicated by the
>LICENSE
>>> file contained therein.

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Clamav for educational institutions ?

2019-04-05 Thread Joel Esler (jesler) via clamav-users 
Correct.  Which is why we recommend people compile from source for full 
functionality.  

Sent from my  iPhone

> On Apr 5, 2019, at 20:12, Scott Kitterman via clamav-users 
>  wrote:
> 
> The unrar stuff is still free to use.
> 
> Due to modification restrictions Debian splits it off into the unofficial 
> non-free repository.
> 
> Scott K
> 
>> On April 6, 2019 12:03:03 AM UTC, "J.R. via clamav-users" 
>>  wrote:
>> I just doubled checked, but I don't see a LICENSE file in the
>> clamav-0.101.2.tar.gz archive???
>> 
>> EDIT - There is the GPLv2 contained in the COPYING file. I just
>> realized each of those files gives the licence for each part of
>> ClamAV. Probably the most notable is the unrar licence, which if I
>> recall RHEL/CentOS disables due to licence conflicts?
>> 
>> 
>> 
>> On Fri, Apr 5, 2019 at 4:30 PM Joel Esler (jesler) 
>> wrote:
>>> 
>>> That’s the content on the website.  ClamAV, the software, is governed
>> by the GPLv2 and other associates licenses as indicated by the LICENSE
>> file contained therein.
>>> 
>> 
>> ___
>> 
>> clamav-users mailing list
>> clamav-users@lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>> 
>> 
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>> 
>> http://www.clamav.net/contact.html#ml
> 
> ___
> 
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml


smime.p7s
Description: S/MIME cryptographic signature

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Clamav for educational institutions ?

2019-04-05 Thread J.R. via clamav-users 
I just doubled checked, but I don't see a LICENSE file in the
clamav-0.101.2.tar.gz archive???

EDIT - There is the GPLv2 contained in the COPYING file. I just
realized each of those files gives the licence for each part of
ClamAV. Probably the most notable is the unrar licence, which if I
recall RHEL/CentOS disables due to licence conflicts?



On Fri, Apr 5, 2019 at 4:30 PM Joel Esler (jesler)  wrote:
>
> That’s the content on the website.  ClamAV, the software, is governed by the 
> GPLv2 and other associates licenses as indicated by the LICENSE file 
> contained therein.
>

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] [External] Re: Scan very slow

2019-04-05 Thread Tim Hawkins 
Hi Micah

Does clamav partition the database so that signatures that are mainly 
associated with email scanning can be dropped out for folks only needing 
filesystems scans,  none of our systems use email, and we dont make use of the 
mailer extension.

Having to load all the email focused signatures could as you have observed 
impact performance.

Sent from Nine

From: "Micah Snyder (micasnyd) via clamav-users" 
Sent: Saturday, April 6, 2019 03:18
To: ClamAV users ML; Mark Allan
Cc: Micah Snyder (micasnyd)
Subject: [External] Re: [clamav-users] Scan very slow

Regarding slow scan times today (and slow scan times in general), it appears 
that the signatures we generate based on PhishTank’s feed for phishing URLs are 
resulting in very slow load and scan times.

Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) 
coinciding with the immediate performance drop for load time and scan time.  
One user reported that the load time today on some of his slower machines was 
slow enough to exceed the timeout for service startup 
(https://bugzilla.clamav.net/show_bug.cgi?id=12317).

In limited testing on my own machine I saw the following change after dropping 
the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:

  *   Database load time on my laptop went from 75.43203997612 seconds down to 
14.859203100204468 seconds
  *   Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.

After some discussion between the teams that work on ClamAV and ClamAV 
signature content and deployment, we’ve agreed to drop PhishTank signatures 
from the database until we can determine a way to craft Phishtank signatures 
without incurring such a significant performance hit.

The daily update tomorrow will have the change.

-Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.



From: clamav-users  on behalf of "Micah 
Snyder (micasnyd) via clamav-users" 
Reply-To: ClamAV users ML 
Date: Friday, April 5, 2019 at 1:08 PM
To: Mark Allan , ClamAV users ML 

Cc: "Micah Snyder (micasnyd)" 
Subject: Re: [clamav-users] Scan very slow

Hi Mark,

Sorry about the delay in responding.  I hadn’t looked at my clamav-users filter 
this morning.  Just investigating now.  Will respond when I know more.

-Micah

From: Mark Allan 
Date: Friday, April 5, 2019 at 9:12 AM
To: ClamAV users ML , "Micah Snyder (micasnyd)" 

Subject: Re: [clamav-users] Scan very slow

Also CC'ing Micah directly as the mailing list would appear to be offline (at 
least lists.clamav.net isn't responding to http 
requests anyway)

It looks like scan times have gone through the roof. As Oya said, they're still 
considerably higher than they were a couple of months ago, but today's scan 
time is insane.

Yesterday's scan using
0.101.2:58:25409:1554370140:1:63:48554:328
took 7m 3s

On the same hardware, scanning the same read-only disk image, with today's scan 
using
0.101.2:58:25410:1554452941:1:63:48557:328
the scan time has jumped to 26m 15s

This is the longest it has ever taken to scan this volume (cf my previous email 
of 25th March)

Is there anything that can be excluded?

Best regards
Mark

On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users 
mailto:clamav-users@lists.clamav.net>> wrote:
Thanks Oya for the update.  We will continue to investigate the signature 
performance issue.

Regards,
Micah

On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" 
mailto:clamav-users-boun...@lists.clamav.net>
 on behalf of oyam...@promark-inc.com> wrote:

Hi Micah

It seems that the  scanning slow down issue of this time has been solved
at some level with CVD Update of the other day.
However, there is still big discrepancy in between the current condition and
the last condition in one month ago.

DateFiles   Scan time
2019/02/15  2550338 08:53:57
2019/03/15  2612792 19:22:54
2019/03/26  2634489 18:13:56
2019/03/27  2637201 18:10:05

We know the improvement of this time is due to the details of CVD, because
we did not make any change on the user's system.
We are going to try some tuning for scanning.

We like to know if you still have some room to make further improvement
for this slow down issue.
Thank you for your help, in advance.

Best regards,
Oya

On Mon, 25 Mar 2019 15:45:02 +
"Micah Snyder \(micasnyd\) via clamav-users" 
mailto:clamav-users@lists.clamav.net>> wrote:

> Hi Mark, all:
>
> I’m disappointed to hear that it is still slow for you.
>
> We found that the target-type of signatures used for PhishTank.Phishing 
signatures were causing a significant slowdown.   We have dropped them as of 
this past Saturday ( https://lists.gt.net/clamav/virusdb/75279 ) and in the 
last two updates have been re-adding them

Re: [clamav-users] Clamav for educational institutions ?

2019-04-05 Thread Joel Esler (jesler) via clamav-users 
That’s the content on the website.  ClamAV, the software, is governed by the 
GPLv2 and other associates licenses as indicated by the LICENSE file contained 
therein.  

Sent from my  iPhone

> On Apr 5, 2019, at 17:18, J.R. via clamav-users 
>  wrote:
> 
> At the bottom of the page on the website it says:
> 
> All content on this website, unless otherwise noted, is licensed under
> the Creative Commons Attribution - NoDerivs License.
> 
> With a link to: https://creativecommons.org/licenses/by-nd/2.5/
> 
> Which says:
> 
> You are free to:
> Share — copy and redistribute the material in any medium or format for
> any purpose, even commercially.
> 
> ___
> 
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml


smime.p7s
Description: S/MIME cryptographic signature

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] Clamav for educational institutions ?

2019-04-05 Thread Timi koli via clamav-users 
Hi Guys,

Does anyone knows if the usage of the clamav for linux is free for
educational institutions or does it have to be a paid one.

I tried to find it on internet but couldn't find it.

Tim,

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Scan very slow

2019-04-05 Thread Micah Snyder (micasnyd) via clamav-users 
Hi Mark,

Sorry about the delay in responding.  I hadn’t looked at my clamav-users filter 
this morning.  Just investigating now.  Will respond when I know more.

-Micah

From: Mark Allan 
Date: Friday, April 5, 2019 at 9:12 AM
To: ClamAV users ML , "Micah Snyder (micasnyd)" 

Subject: Re: [clamav-users] Scan very slow

Also CC'ing Micah directly as the mailing list would appear to be offline (at 
least lists.clamav.net isn't responding to http 
requests anyway)

It looks like scan times have gone through the roof. As Oya said, they're still 
considerably higher than they were a couple of months ago, but today's scan 
time is insane.

Yesterday's scan using
0.101.2:58:25409:1554370140:1:63:48554:328
took 7m 3s

On the same hardware, scanning the same read-only disk image, with today's scan 
using
0.101.2:58:25410:1554452941:1:63:48557:328
the scan time has jumped to 26m 15s

This is the longest it has ever taken to scan this volume (cf my previous email 
of 25th March)

Is there anything that can be excluded?

Best regards
Mark

On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users 
mailto:clamav-users@lists.clamav.net>> wrote:
Thanks Oya for the update.  We will continue to investigate the signature 
performance issue.

Regards,
Micah

On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" 
mailto:clamav-users-boun...@lists.clamav.net>
 on behalf of oyam...@promark-inc.com> wrote:

Hi Micah

It seems that the  scanning slow down issue of this time has been solved
at some level with CVD Update of the other day.
However, there is still big discrepancy in between the current condition and
the last condition in one month ago.

DateFiles   Scan time
2019/02/15  2550338 08:53:57
2019/03/15  2612792 19:22:54
2019/03/26  2634489 18:13:56
2019/03/27  2637201 18:10:05

We know the improvement of this time is due to the details of CVD, because
we did not make any change on the user's system.
We are going to try some tuning for scanning.

We like to know if you still have some room to make further improvement
for this slow down issue.
Thank you for your help, in advance.

Best regards,
Oya

On Mon, 25 Mar 2019 15:45:02 +
"Micah Snyder \(micasnyd\) via clamav-users" 
mailto:clamav-users@lists.clamav.net>> wrote:

> Hi Mark, all:
>
> I’m disappointed to hear that it is still slow for you.
>
> We found that the target-type of signatures used for PhishTank.Phishing 
signatures were causing a significant slowdown.   We have dropped them as of 
this past Saturday ( https://lists.gt.net/clamav/virusdb/75279 ) and in the 
last two updates have been re-adding them with more specific scan target types. 
 We’re now investigating some other optimizations we can make for the next 
major ClamAV release to improve scan times but at present we don’t have any 
other leads for signatures that may be slowing down scans.
>
> Regards,
> Micah
>
>
> From: clamav-users 
mailto:clamav-users-boun...@lists.clamav.net>>
 on behalf of Mark Allan via clamav-users 
mailto:clamav-users@lists.clamav.net>>
> Reply-To: ClamAV users ML 
mailto:clamav-users@lists.clamav.net>>
> Date: Monday, March 25, 2019 at 9:37 AM
> To: ClamAV users ML 
mailto:clamav-users@lists.clamav.net>>
> Cc: Mark Allan mailto:markjal...@gmail.com>>
> Subject: Re: [clamav-users] Scan very slow
>
> Cheers Steve,
>
> In the interest of completeness, here's the scan from today (TXT from 
DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement 
in scan time, although at 6m 7s it's still almost twice what it used to be.
>
> Mark
>
> On Mon, 25 Mar 2019 at 12:56, Steve Basford 
mailto:steveb_cla...@sanesecurity.com>>>
 wrote:
> On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > Hi all,
> >
> te.
> >
> > Hopefully this helps someone to narrow things down a bit.
> >
> > Mark
> >
>
> 18/3/19 10m 49s TXT from DNS:
> 0.101.1:58:25392:1552904941:1:63:48507:328  ***
>
> Here's the changes for the above update:
>
> https://lists.gt.net/clamav/virusdb/75154
>
> You can also check sigs quickly per update:
>
> https://lists.gt.net/clamav/virusdb/
>
>
>
> --
> Cheers,
>
> Steve
> Twitter: @sanesecurity
>
> ___
>
> clamav-users mailing list
> 
clamav-users@lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a

Re: [clamav-users] Malformed pattern daily.ldb version 25410

2019-04-05 Thread David Raynor 
I can recreate that same issue with daily cvd 25410, using ClamAV 0.100.1.
That was the first 0.100.X I had handy to do a quick test.
The problem is something specific to sigtool and only the list-sigs
feature. It does not affect clamscan or clamd, and does not affect the
--find-sigs option of sigtool.
We do ongoing signature load testing with several different versions of
ClamAV, but focus on scan testing.

It does still happen with the latest release so I'll talk with the team
about opening this as a bug.

Thanks for the report.

Dave R.

On Fri, Apr 5, 2019 at 11:12 AM David Shrimpton via clamav-users <
clamav-users@lists.clamav.net> wrote:

> I can reproduce the Malformed pattern problem with a file with just the
> one  signature:
>
> Xls.Downloader.Powload-6923120-0 which is an even longer one .
>
> This is 4 signatures before Doc.Trojan.Agent-6923124-0 in daily.ldb
>
> sigtool reports the wrong line numbering eg with a file with just
> Xls.Downloader.Powload-6923120-0 it reports
> the problem as being on line 2.  It seems to be 4 lines out when reporting
> on the whole daily.ldb
>
> again sigtool --find Xls.Downloader.Powload-6923120-0  | sigtool
> --decode-sigs
>
> doesn't show a problem.
>
> clamscan --debug -d file_with_just_the_sig_above.ldb somefile
> doesn't show a problem.
>
> Xls.Downloader.Powload-6923120-0 turned up in daily 25410 which was when
> the problem started
>
> Maybe sigtool --list can't handle long signatures in ClamAV 0.100.2
>
> There does seem a pointlessness to signatures based upon exact variable
> names etc that are obfuscated
> and  likely will vary with each sample.  A regex signature to get any
> variable name would be better.
>
>
> David Shrimpton
>
> 
> From: clamav-users  on behalf of
> Arnaud Jacques 
> Sent: Saturday, April 6, 2019 12:27 AM
> To: clamav-users@lists.clamav.net
> Subject: Re: [clamav-users] Malformed pattern daily.ldb version 25410
>
> Hello,
>
> > sigtool --find-sigs Doc.Trojan.Agent-6923124-0 | sigtool --decode-sigs
> I don't understand why this signature is so long, and why it is based on
> always changing variables.
>
>
> ___
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>


-- 
---
Dave Raynor
Talos Security Intelligence and Research Group
dray...@sourcefire.com

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Malformed pattern daily.ldb version 25410

2019-04-05 Thread David Shrimpton via clamav-users 
I can reproduce the Malformed pattern problem with a file with just the one  
signature:

Xls.Downloader.Powload-6923120-0 which is an even longer one .

This is 4 signatures before Doc.Trojan.Agent-6923124-0 in daily.ldb

sigtool reports the wrong line numbering eg with a file with just 
Xls.Downloader.Powload-6923120-0 it reports
the problem as being on line 2.  It seems to be 4 lines out when reporting on 
the whole daily.ldb

again sigtool --find Xls.Downloader.Powload-6923120-0  | sigtool --decode-sigs 

doesn't show a problem.

clamscan --debug -d file_with_just_the_sig_above.ldb somefile
doesn't show a problem.

Xls.Downloader.Powload-6923120-0 turned up in daily 25410 which was when the 
problem started

Maybe sigtool --list can't handle long signatures in ClamAV 0.100.2

There does seem a pointlessness to signatures based upon exact variable names 
etc that are obfuscated
and  likely will vary with each sample.  A regex signature to get any variable 
name would be better.


David Shrimpton


From: clamav-users  on behalf of Arnaud 
Jacques 
Sent: Saturday, April 6, 2019 12:27 AM
To: clamav-users@lists.clamav.net
Subject: Re: [clamav-users] Malformed pattern daily.ldb version 25410

Hello,

> sigtool --find-sigs Doc.Trojan.Agent-6923124-0 | sigtool --decode-sigs
I don't understand why this signature is so long, and why it is based on
always changing variables.


___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Malformed pattern daily.ldb version 25410

2019-04-05 Thread Arnaud Jacques 

Hello,


sigtool --find-sigs Doc.Trojan.Agent-6923124-0 | sigtool --decode-sigs
I don't understand why this signature is so long, and why it is based on 
always changing variables.


--
Cordialement / Best regards,

Arnaud Jacques
Gérant de SecuriteInfo.com

Téléphone : +33-(0)3.44.39.76.46
E-mail : a...@securiteinfo.com
Site web : https://www.securiteinfo.com
Facebook : https://www.facebook.com/pages/SecuriteInfocom/132872523492286
Twitter : @SecuriteInfoCom

Securiteinfo.com
La Sécurité Informatique - La Sécurité des Informations.
266, rue de Villers
60123 Bonneuil en Valois


___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[clamav-users] Malformed pattern daily.ldb version 25410

2019-04-05 Thread David Shrimpton via clamav-users 
sigtool --list=/path/daily.cld

is returning:

ERROR: listdb: Error listing database 
/tmp/clamav-0348baa027819612194d4bd1d7aed9d0.tmp/daily.ldb
ERROR: listdb: Malformed pattern line 52912 (file 
/tmp/clamav-0348baa027819612194d4bd1d7aed9d0.tmp/daily.ldb)

extracting daily.ldb with sigtool --unpack-current=daily  and finding line 
52912 with sed shows:

sed -ne '52912p' daily.ldb
Doc.Trojan.Agent-6923124-0;Engine:51-255,Target:2;0&1&2&3&4&5&6&7;424f617547596379616f6b795a69614d5563795375646167496245745968203d2053776974636828424f617547596379616f6b795a69614d5563795375646167496245745968203c20312c20227a4f614f706962796175434966416a6968416469776f7c20424f617547596379616f6b795a69614d5563795375646167496245745968203e20322c2022446957415657596b59466578222c20424f617547596379616f6b795a69614d5563795375646167496245745968203c20352c2043566566595265535576596e4f44416a4879526f58202b2022716f674f5179565645485547656d45662229;4759487a6174695a4150414a79724f774f76554661534174495a6167596b654d41647962416a6a5973203d20537769746368284759487a6174695a4150414a79724f774f76554661534174495a6167596b654d41647962416a6a5973203c20342c20224c4978756741616543497844494e45222c204759487a6174695a4150414a79724f774f76554661534174495a6167596b654d41647962416a6a5973203e20322c202248454778414e6954614a222c204759487a6174695a4150414a79724f774f76554661534174495a6167596b654d41647962416a6a5973203c20342c2074694655644f6
 
3794e6942757645436b596a61506168202b2022425967754e45744f776f546f4d6543692229;48594455775557555055775574796165685574203d20435661722835202b2038202b203329;486f757220226c614e5567657a4172615a6943497261636922202b202268686f6875776b494122;4c4f534f4e596261516f62203d20435661722839202b2035202b20313029;6945786d6f545577555643614c597679584f62455441706567754b59524f78203d2032;72697a6f76414a41486f68654665487570754265764944656375454379203d2038;7a6948557141444976616d5969557759786157654e6947556f4741446157614475526f56754865203d2039

cut and paste has added a line break above but this sig looks OK on first glance

sigtool --find-sigs Doc.Trojan.Agent-6923124-0 | sigtool --decode-sigs
shows sensible output for the above signature, so I am not sure this is the 
exact one causing the sigtool error.

The problem started from database version 25410 upgrade , so it appears one (or 
more) sigs are Malformed in 25410

ClamAV 0.100.2/25410/Fri Apr  5 17:58:26 2019


David Shrimpton


___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Updating multiple servers

2019-04-05 Thread Bowie Bailey 
On 4/4/2019 9:01 PM, Tim Hawkins wrote:
> We have a large number of services running inside kubernetes that need to have
> access to clamav,  given the sheer number, i dont want to have to run 
> freshclam
> process on each virtual machine (container), due to the managemeht and 
> monitoring
> overhead, and the risk of some not updating for variouis reasons.
>
> Is there any easy way i can share the directories containing the definition
> database on one server image to all the others so i only have one machine to
> monitor updates on, we can use docker.kubernetes ability to share persistent
> volumes to do this,  we will be running clamav in single file scan mode, and 
> wont
> be using the daemon, so syncronising restart of the daemon on updates is not 
> required.

If you are simply scanning single files and loading the databases every time, 
then
you should be able to share the database directory with whatever method you have
available.

On the other hand, keep in mind that it can take time for clamscan to load the
databases (especially for slower systems or if you have lots of third-party
signatures).  If you have any volume at all, you may want to use the daemon 
instead
since it is MUCH faster.  One solution would be to run the daemon on one server 
and
open a TCP port so the other servers can connect to it with clamdscan to do 
scans. 
That way you only have one database directory and one daemon process to worry 
about.

-- 
Bowie

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Scan very slow

2019-04-05 Thread Joel Esler (jesler) via clamav-users 

> On Apr 5, 2019, at 09:13, Mark Allan via clamav-users 
>  wrote:
> 
> Also CC'ing Micah directly as the mailing list would appear to be offline (at 
> least lists.clamav.net isn't responding to http requests anyway

May want to try https.

smime.p7s
Description: S/MIME cryptographic signature

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Re: [clamav-users] Scan very slow

2019-04-05 Thread Mark Allan via clamav-users 
Also CC'ing Micah directly as the mailing list would appear to be offline
(at least lists.clamav.net isn't responding to http requests anyway)

It looks like scan times have gone through the roof. As Oya said, they're
still considerably higher than they were a couple of months ago, but
today's scan time is insane.

Yesterday's scan using
0.101.2:58:25409:1554370140:1:63:48554:328
took 7m 3s

On the same hardware, scanning the same read-only disk image, with today's
scan using
0.101.2:58:25410:1554452941:1:63:48557:328
the scan time has jumped to 26m 15s

This is the longest it has ever taken to scan this volume (cf my previous
email of 25th March)

Is there anything that can be excluded?

Best regards
Mark

On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <
clamav-users@lists.clamav.net> wrote:

> Thanks Oya for the update.  We will continue to investigate the signature
> performance issue.
>
> Regards,
> Micah
>
> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <
> clamav-users-boun...@lists.clamav.net on behalf of oyam...@promark-inc.com>
> wrote:
>
> Hi Micah
>
> It seems that the  scanning slow down issue of this time has been
> solved
> at some level with CVD Update of the other day.
> However, there is still big discrepancy in between the current
> condition and
> the last condition in one month ago.
>
> DateFiles   Scan time
> 2019/02/15  2550338 08:53:57
> 2019/03/15  2612792 19:22:54
> 2019/03/26  2634489 18:13:56
> 2019/03/27  2637201 18:10:05
>
> We know the improvement of this time is due to the details of CVD,
> because
> we did not make any change on the user's system.
> We are going to try some tuning for scanning.
>
> We like to know if you still have some room to make further improvement
> for this slow down issue.
> Thank you for your help, in advance.
>
> Best regards,
> Oya
>
> On Mon, 25 Mar 2019 15:45:02 +
> "Micah Snyder \(micasnyd\) via clamav-users" <
> clamav-users@lists.clamav.net> wrote:
>
> > Hi Mark, all:
> >
> > I’m disappointed to hear that it is still slow for you.
> >
> > We found that the target-type of signatures used for
> PhishTank.Phishing signatures were causing a significant slowdown.   We
> have dropped them as of this past Saturday (
> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates
> have been re-adding them with more specific scan target types.  We’re now
> investigating some other optimizations we can make for the next major
> ClamAV release to improve scan times but at present we don’t have any other
> leads for signatures that may be slowing down scans.
> >
> > Regards,
> > Micah
> >
> >
> > From: clamav-users  on
> behalf of Mark Allan via clamav-users 
> > Reply-To: ClamAV users ML 
> > Date: Monday, March 25, 2019 at 9:37 AM
> > To: ClamAV users ML 
> > Cc: Mark Allan 
> > Subject: Re: [clamav-users] Scan very slow
> >
> > Cheers Steve,
> >
> > In the interest of completeness, here's the scan from today (TXT
> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked
> improvement in scan time, although at 6m 7s it's still almost twice what it
> used to be.
> >
> > Mark
> >
> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <
> steveb_cla...@sanesecurity.com>
> wrote:
> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > > Hi all,
> > >
> > te.
> > >
> > > Hopefully this helps someone to narrow things down a bit.
> > >
> > > Mark
> > >
> >
> > 18/3/19 10m 49s TXT from DNS:
> > 0.101.1:58:25392:1552904941:1:63:48507:328  ***
> >
> > Here's the changes for the above update:
> >
> > https://lists.gt.net/clamav/virusdb/75154
> >
> > You can also check sigs quickly per update:
> >
> > https://lists.gt.net/clamav/virusdb/
> >
> >
> >
> > --
> > Cheers,
> >
> > Steve
> > Twitter: @sanesecurity
> >
> > ___
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>
>
>
> ___
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
>