Re: [clamav-users] Extremely slow PDF file scanning

[Nikolay Belaevski via clamav-users]

Hi Micah,

Thank you very much for your attention to this matter! I have re-shared files; 
original file links should be working now:

File: https://storage.googleapis.com/upload-samples/Museum_26MB.pdf
Config file: https://storage.googleapis.com/upload-samples/clamd.conf
Debug log: https://storage.googleapis.com/upload-samples/debug.log
Extract file size data: https://storage.googleapis.com/upload-samples/files.log

I will try patched versions when they are available.

Best regards,
  Nikolay


From: Micah Snyder (micasnyd) 
Date: Tuesday, November 2, 2021 at 17:30
To: ClamAV users ML 
Cc: Nikolay Belaevski 
Subject: Re: [clamav-users] Extremely slow PDF file scanning
Hi Nikolay,

Sorry this slipped by me.  I'd be happy to take a look at the PDF you were 
having scan speed issues with.  I see that it's no longer available with the 
URL you originally provided. If you could share it again, I'll spend some time 
with it to try to see what's going on.

As a heads up, we have couple patch versions coming out tomorrow which I hope 
will show some scan time improvements and detection improvements as a result of 
work overhauling some scan recursion and embedded file type detection logic. I 
don't expect it will help in this particular case, but ... maybe!  *shrugs*

Regards,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.

From: clamav-users  on behalf of Nikolay 
Belaevski via clamav-users 
Sent: Monday, October 4, 2021 6:03 PM
To: ClamAV users ML 
Cc: Nikolay Belaevski 
Subject: Re: [clamav-users] Extremely slow PDF file scanning


Any feedback from ClamAV developers, please: should I open a defect for the 
problem or is it expected that PDF file scanning takes few minutes?



Thanks,

  Nikolay



From: Nikolay Belaevski 
Date: Saturday, September 25, 2021 at 11:32
To: ClamAV users ML 
Subject: Re: [clamav-users] Extremely slow PDF file scanning

Hi Ged,



Thank you for your response! You are right, the configuration below is not 
secure and thanks again for pointing to this! For production we’ll use 
different configuration that simply issues alert for the provided file. The one 
attached here is more a less copy of default configuration with the limits 
increased to extremely high values in attempt to get scanning completed and 
then try to tweak them down to see what exactly is causing the alert on the 
file.



Release notes for 0.104.0 mention “Fixed bytecode match evaluation for PDF 
bytecode hooks in PDF file scans.”. Looks like something that’s been fixed, 
yes. For curiosity I have tried to disable bytecode and the scanning completes 
faster of course, but it must be enabled for best results AFAIK.



Looking at the total size of temporary files created, I see that scan has 
produced 1.7Gb of temporary files extracted from PDF and that may explain why 
scanning takes that much time. For comparison, when I extract all images from 
the file using pdfimages program, I’m seeing about 21Mb. And another 
comparison, when I use pdftk to concatenate original file with the simple 
one-page text file and scan result, scan takes just two seconds and the size of 
temporary files produced is about 6Mb. I.e. for a slightly larger input scan 
time is much better! That’s the reason I’m suspecting there may be something in 
the original file that confuses ClamAV parser / analyzer and would be 
interesting for ClamAV developers to check. On the side note, I have tried few 
large PDF files from Google scanned library, no issues at all and they are 
scanned quickly.



Best regards,

  Nikolay



From: clamav-users  on behalf of G.W. 
Haywood via clamav-users 
Date: Saturday, September 25, 2021 at 10:44
To: Nikolay Belaevski via clamav-users 
Cc: G.W. Haywood 
Subject: Re: [clamav-users] Extremely slow PDF file scanning

Hi there,

On Fri, 24 Sep 2021, Nikolay Belaevski via clamav-users wrote:

> Iʼm investigating why it takes about five minutes for ClamAV 0.104.0
> to scan PDF file. Can someone help me, please?

There's a note in NEWS.md that a fault in PDF scanning was fixed in
version 0.104.  I don't know if it's relevant but it might be worth a
look.  More importantly did you read the warnings in the documentation
about the settings that you've changed?

Snippets from 'clamconf -n' reporting about your clamd.conf, with my
comments added below each item:

8<--

TCPSocket = "3310"
# TCP sockets are unprotected from remote access.  Careful!

DisableCache = "yes"
# This will cause clamd to scan the file every time it sees it, and not
# to use its internal caching.

Foreground = "yes"
# I guess there's reason for this?

Debug = "yes"
# This may affect performance, but I would not expect gross effects.

MaxScanTime = "60"
# Ten minutes!  *Fifty* times the default of 12 seconds.  According to
# the man page, excessive times can cause DOS conditions.

MaxScanSize = "4194304000"
# 4GBytes.  *Forty* 

Re: [clamav-users] Extremely slow PDF file scanning

[Micah Snyder (micasnyd) via clamav-users]

Hi Nikolay,

Sorry this slipped by me.  I'd be happy to take a look at the PDF you were 
having scan speed issues with.  I see that it's no longer available with the 
URL you originally provided. If you could share it again, I'll spend some time 
with it to try to see what's going on.

As a heads up, we have couple patch versions coming out tomorrow which I hope 
will show some scan time improvements and detection improvements as a result of 
work overhauling some scan recursion and embedded file type detection logic. I 
don't expect it will help in this particular case, but ... maybe!  *shrugs*

Regards,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.

From: clamav-users  on behalf of Nikolay 
Belaevski via clamav-users 
Sent: Monday, October 4, 2021 6:03 PM
To: ClamAV users ML 
Cc: Nikolay Belaevski 
Subject: Re: [clamav-users] Extremely slow PDF file scanning


Any feedback from ClamAV developers, please: should I open a defect for the 
problem or is it expected that PDF file scanning takes few minutes?



Thanks,

  Nikolay



From: Nikolay Belaevski 
Date: Saturday, September 25, 2021 at 11:32
To: ClamAV users ML 
Subject: Re: [clamav-users] Extremely slow PDF file scanning

Hi Ged,



Thank you for your response! You are right, the configuration below is not 
secure and thanks again for pointing to this! For production we’ll use 
different configuration that simply issues alert for the provided file. The one 
attached here is more a less copy of default configuration with the limits 
increased to extremely high values in attempt to get scanning completed and 
then try to tweak them down to see what exactly is causing the alert on the 
file.



Release notes for 0.104.0 mention “Fixed bytecode match evaluation for PDF 
bytecode hooks in PDF file scans.”. Looks like something that’s been fixed, 
yes. For curiosity I have tried to disable bytecode and the scanning completes 
faster of course, but it must be enabled for best results AFAIK.



Looking at the total size of temporary files created, I see that scan has 
produced 1.7Gb of temporary files extracted from PDF and that may explain why 
scanning takes that much time. For comparison, when I extract all images from 
the file using pdfimages program, I’m seeing about 21Mb. And another 
comparison, when I use pdftk to concatenate original file with the simple 
one-page text file and scan result, scan takes just two seconds and the size of 
temporary files produced is about 6Mb. I.e. for a slightly larger input scan 
time is much better! That’s the reason I’m suspecting there may be something in 
the original file that confuses ClamAV parser / analyzer and would be 
interesting for ClamAV developers to check. On the side note, I have tried few 
large PDF files from Google scanned library, no issues at all and they are 
scanned quickly.



Best regards,

  Nikolay



From: clamav-users  on behalf of G.W. 
Haywood via clamav-users 
Date: Saturday, September 25, 2021 at 10:44
To: Nikolay Belaevski via clamav-users 
Cc: G.W. Haywood 
Subject: Re: [clamav-users] Extremely slow PDF file scanning

Hi there,

On Fri, 24 Sep 2021, Nikolay Belaevski via clamav-users wrote:

> Iʼm investigating why it takes about five minutes for ClamAV 0.104.0
> to scan PDF file. Can someone help me, please?

There's a note in NEWS.md that a fault in PDF scanning was fixed in
version 0.104.  I don't know if it's relevant but it might be worth a
look.  More importantly did you read the warnings in the documentation
about the settings that you've changed?

Snippets from 'clamconf -n' reporting about your clamd.conf, with my
comments added below each item:

8<--

TCPSocket = "3310"
# TCP sockets are unprotected from remote access.  Careful!

DisableCache = "yes"
# This will cause clamd to scan the file every time it sees it, and not
# to use its internal caching.

Foreground = "yes"
# I guess there's reason for this?

Debug = "yes"
# This may affect performance, but I would not expect gross effects.

MaxScanTime = "60"
# Ten minutes!  *Fifty* times the default of 12 seconds.  According to
# the man page, excessive times can cause DOS conditions.

MaxScanSize = "4194304000"
# 4GBytes.  *Forty* times the default of 100M, and in the clamd.conf
# man page there is a dire warning about setting this limit too high.
# I'm also not sure how the absolute 2G limit on file size impinges.

MaxFileSize = "52428800"
# Twice the default.  Another dire warning in the man page.

MaxFiles = "5"
# Five times the default.  Dire warning.

PCREMaxFileSize = "104857600"
# Four times the default.  Specific warning about performance.

8<--

I think you need to look carefully at the configuration changes which
you've made, perhaps do some testing to establish whether your system
can support scanning with those 

Re: [clamav-users] Missing Mac OS .pkg installer

[Micah Snyder (micasnyd) via clamav-users]

The macOS installer PKG for 0.104.0 should once again be available on 
https://www.clamav.net/downloads.  Thanks for reporting the issue.  Note there 
is no GPG .sig file for the macOS PKG because the PKG is itself signed and 
notarized.
Download - ClamAVNet
Immunet, powered by ClamAV, is a fast, fully featured Windows desktop 
anti-virus (AV) solution that utilizes the power of advanced cloud-based 
detection techniques and the strength of the time-tested ClamAV engine.
www.clamav.net

Regards,
Micah


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.

From: clamav-users  on behalf of Micah 
Snyder (micasnyd) via clamav-users 
Sent: Sunday, October 31, 2021 12:46 PM
To: ClamAV users ML 
Cc: Micah Snyder (micasnyd) 
Subject: Re: [clamav-users] Missing Mac OS .pkg installer

I think maybe there is an issue on the website. I believe Joel is saying the 
macOS installer PKG should appear under "macOS" in the "alternate versions of 
ClamAV" section.  However, it does seem to be missing.
I'll check with the web team about it.


Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.

From: clamav-users  on behalf of Joel 
Esler (jesler) via clamav-users 
Sent: Friday, October 29, 2021 3:46 AM
To: ClamAV users ML 
Cc: Joel Esler (jesler) 
Subject: Re: [clamav-users] Missing Mac OS .pkg installer

https://www.clamav.net/downloads

Scroll down to “alternate versions of ClamAV” and click on macOS.

—
Sent from my  iPhone

On Oct 28, 2021, at 13:40, Vaughn A. Hart  wrote:


Hi Team Clamav,

In your documentsation you state that there is a pkg installer for Mac OS that 
supports Intel and M1 but I can't find it on your download page.

Sincerely,

-Vaughn

--


Vaughn A. Hart
General Manager
Aegis IT, LLC
646-284-4291
vau...@aegisitnyc.com
https://www.linkedin.com/in/vahart

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

___

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml