Bug#904111: clamav-daemon causing deadlocks/blocking I/O
Ah, so I think you may have the winner. I set my temp directory to be something other than /tmp, and turned ClamAV back on, and it's been running for about an hour now with no obvious ill effects. I will report back if something else crops up, but I think this may solve it. Thank you! On Mon, Nov 19, 2018 at 2:31 PM Sebastian Andrzej Siewior wrote: > On 2018-11-19 21:01:07 [+0100], To Adam Lambert wrote: > > On 2018-11-12 10:17:32 [-0800], Adam Lambert wrote: > > > I believe I already supplied all that way back when I opened up this > bug > > > report. But for reference, here it is again: > > > > I tried it back then with no luck. Thanks for the info. I will try to > > reproduce this asap and get back to you. > > Okay. It triggers. This > > OnAccessIncludePath /tmp > > seems to be the root of all evil. Removing this option or adding > > TemporaryDirectory /var/tmp/ > > seems to make it go away. So I *think* the problem is that clamav makes > temporary files during scanning which in turn it tries to scan and > blocks itself. > Can you acknowledge the behaviour? > > Sebastian >
Bug#904111: clamav-daemon causing deadlocks/blocking I/O
I believe I already supplied all that way back when I opened up this bug report. But for reference, here it is again: 1) Standard kernel boot params that come after a vanilla Debian install (ie: I have not modified them). 2) Config file is below. All I "do" is 'service clamav-daemon start' and wait about 90 seconds and the system is unresponsive. This seems to be related to the scan-on-access feature doing blocking I/O/deadlocking in some way. I can speed up the crash by doing something like 'cat ~/somefile > /dev/null' or otherwise reading files in one of the ScanOnAccess folders. Clamd.conf is cut/pasted below: # -- begin LocalSocket /var/run/clamav/clamd.ctl FixStaleSocket true LocalSocketGroup clamav LocalSocketMode 666 #PreludeEnable no #PreludeAnalyzerName ClamAV # TemporaryDirectory is not set to its default /tmp here to make overriding # the default with environment variables TMPDIR/TMP/TEMP possible User root ScanMail true ScanArchive true ArchiveBlockEncrypted false MaxDirectoryRecursion 15 FollowDirectorySymlinks false FollowFileSymlinks false ReadTimeout 180 MaxThreads 12 MaxConnectionQueueLength 15 LogSyslog true LogRotate true LogFacility LOG_LOCAL6 LogClean false LogVerbose false DatabaseDirectory /var/lib/clamav OfficialDatabaseOnly false SelfCheck 3600 Foreground false Debug false ScanPE true MaxEmbeddedPE 10M ScanOLE2 true ScanPDF true ScanHTML true MaxHTMLNormalize 10M MaxHTMLNoTags 2M MaxScriptNormalize 5M MaxZipTypeRcg 1M ScanSWF true DetectBrokenExecutables false ExitOnOOM false LeaveTemporaryFiles false AlgorithmicDetection true ScanELF true IdleTimeout 30 CrossFilesystems true PhishingSignatures true PhishingScanURLs true PhishingAlwaysBlockSSLMismatch false PhishingAlwaysBlockCloak false PartitionIntersection false DetectPUA false ScanPartialMessages false HeuristicScanPrecedence false StructuredDataDetection false CommandReadTimeout 5 SendBufTimeout 200 MaxQueue 100 ExtendedDetectionInfo true OLE2BlockMacros false # customized ScanOnAccess true #OnAccessMaxFileSize 5M OnAccessPrevention true OnAccessIncludePath /tmp OnAccessIncludePath /home OnAccessIncludePath /root # end customized AllowAllMatchScan true ForceToDisk false DisableCertCheck false DisableCache false MaxScanSize 100M MaxFileSize 25M MaxRecursion 16 MaxFiles 1 MaxPartitions 50 MaxIconsPE 100 PCREMatchLimit 1 PCRERecMatchLimit 5000 PCREMaxFileSize 25M ScanXMLDOCS true ScanHWP3 true MaxRecHWP3 16 StatsEnabled false StatsPEDisabled true StatsHostID auto StatsTimeout 10 StreamMaxLength 25M LogFile /var/log/clamav/clamav.log LogTime true LogFileUnlock false LogFileMaxSize 0 Bytecode true BytecodeSecurity TrustSigned BytecodeTimeout 6 On Sat, Nov 10, 2018 at 12:03 PM Sebastian Andrzej Siewior wrote: > On 2018-11-08 15:15:57 [-0800], Adam Lambert wrote: > > What do you need me to do to provide debug info on this? > I would like to reproduce this. I would need the clamd.conf, kernel > command line if something non-standard and what it is you do. > > If I can reproduce this on my Stretch VM then I try to forward this > upstream or look myself. > > > Thanks, > > Sebastian >
Bug#904111: clamav-daemon causing deadlocks/blocking I/O
I apologize for weighing in late, I saw earlier in the thread that Marc Dequènes reported reproducing it and assumed that would be sufficient. No, this is not solved. I just apt upgrade'd to the latest version (0.100.2+dfsg-0+deb9u1), and again, within seconds, the system went down hard. What do you need me to do to provide debug info on this? And this is indeed a 'critical' level bug - it renders ClamAV (and the underlying system) entirely unusable in any of the 0.100.xxx versions I've tried. Thanks, On Thu, Nov 8, 2018 at 2:28 PM Sebastian Andrzej Siewior wrote: > On 2018-11-03 17:11:07 [+], Scott Kitterman wrote: > > Does anyone still have this problem with 0.100.2? It's been out awhile > and this bug has gone quiet. > > I would suggest to close it. I never had any luck to reproduce it. It > may or may not be a problem but without any additional help to get a > reproducer there is nothing that we can do to either fix it ourself or > throw at upstream. > I'm not sure if severity `critical' applies here after all. > > > Scott K > > Sebastian >
Bug#904111: clamav-daemon causing deadlocks/blocking I/O.
This is my primary workstation, which is not very convenient to test with at this time (I lost 3 hours of work already getting it stabilized again). Could you perhaps use my config on one of your test systems and try to duplicate first? If you can not duplicate, I will be willing to put some more effort into testing on my end. Thanks, On Thu, Jul 19, 2018 at 2:06 PM, Sebastian Andrzej Siewior < sebast...@breakpoint.cc> wrote: > On 2018-07-19 13:38:04 [-0700], Adam Lambert wrote: > > clamd (28514): Using fanotify permission checks may lead to deadlock; > tainting kernel > > and shortly thereafter > > This seems to become true. > > > INFO: task clamd:28512 blocked for more than 120 seconds. > > That is deadlock that happens. > > > I downgraded to 0.99.4+dfsg-1+deb9u1 and system remains stable as it had > been heretofore. > interresting. > > > I suspect this is related to my use of ScanOnAccess true, but not sure. > I think that causes the problem. Could you try to switch it off? > Do you use clamav / the machine for something like a mailserver or so? > > Sebastian >
Bug#904111: clamav-daemon causing deadlocks/blocking I/O.
Package: clamav-daemon Version: 0.100.0+dfsg-0+deb9u2 Severity: critical Justification: breaks the whole system Dear Maintainer, After a recent apt upgrade, within a few minutes, my system started locking up. A reboot would buy me about 2 minutes of working time before it locked up again. I noted the following in the logs that seemed to correspond: clamd (28514): Using fanotify permission checks may lead to deadlock; tainting kernel and shortly thereafter INFO: task clamd:28512 blocked for more than 120 seconds. This seemed to be causing some kind of deadlock as described in the first error, since other programs would go into forever wait mode waiting on I/O (ie: blocking I/O). The other programs could not be kill -9'd. service clamav-daemon stop == system instantly returned to stability. I downgraded to 0.99.4+dfsg-1+deb9u1 and system remains stable as it had been heretofore. I suspect this is related to my use of ScanOnAccess true, but not sure. The only thing I think that is otherwise unusual about my system is that I do not use SystemD nor any major GUI environment (simple IceWM setup). Otherwise, I run a pretty stripped down setup, with as few running processes as possible. I have already downgraded, so you may see incorrectly some versions in the included data of 0.99.4+dfsg-1+deb9u1. 0.99.4+dfsg-1+deb9u1 is the stable version. It is the 0.100.0+dfsg-0+deb9u2 version that is broken. -- Package-specific info: --- configuration --- Checking configuration files in /etc/clamav Config file: clamd.conf --- LogFile = "/var/log/clamav/clamav.log" StatsHostID = "auto" StatsEnabled disabled StatsPEDisabled = "yes" StatsTimeout = "10" LogFileUnlock disabled LogFileMaxSize = "4294967295" LogTime = "yes" LogClean disabled LogSyslog = "yes" LogFacility = "LOG_LOCAL6" LogVerbose disabled LogRotate = "yes" ExtendedDetectionInfo = "yes" PidFile disabled TemporaryDirectory disabled DatabaseDirectory = "/var/lib/clamav" OfficialDatabaseOnly disabled LocalSocket = "/var/run/clamav/clamd.ctl" LocalSocketGroup = "clamav" LocalSocketMode = "666" FixStaleSocket = "yes" TCPSocket disabled TCPAddr disabled MaxConnectionQueueLength = "15" StreamMaxLength = "26214400" StreamMinPort = "1024" StreamMaxPort = "2048" MaxThreads = "12" ReadTimeout = "180" CommandReadTimeout = "5" SendBufTimeout = "200" MaxQueue = "100" IdleTimeout = "30" ExcludePath disabled MaxDirectoryRecursion = "15" FollowDirectorySymlinks disabled FollowFileSymlinks disabled CrossFilesystems = "yes" SelfCheck = "3600" DisableCache disabled VirusEvent disabled ExitOnOOM disabled AllowAllMatchScan = "yes" Foreground disabled Debug disabled LeaveTemporaryFiles disabled User = "root" Bytecode = "yes" BytecodeSecurity = "TrustSigned" BytecodeTimeout = "6" BytecodeUnsigned disabled BytecodeMode = "Auto" DetectPUA disabled ExcludePUA disabled IncludePUA disabled AlgorithmicDetection = "yes" ScanPE = "yes" ScanELF = "yes" DetectBrokenExecutables disabled ScanMail = "yes" ScanPartialMessages disabled PhishingSignatures = "yes" PhishingScanURLs = "yes" PhishingAlwaysBlockCloak disabled PhishingAlwaysBlockSSLMismatch disabled PartitionIntersection disabled HeuristicScanPrecedence disabled StructuredDataDetection disabled StructuredMinCreditCardCount = "3" StructuredMinSSNCount = "3" StructuredSSNFormatNormal = "yes" StructuredSSNFormatStripped disabled ScanHTML = "yes" ScanOLE2 = "yes" OLE2BlockMacros disabled ScanPDF = "yes" ScanSWF = "yes" ScanXMLDOCS = "yes" ScanHWP3 = "yes" ScanArchive = "yes" ArchiveBlockEncrypted disabled ForceToDisk disabled MaxScanSize = "104857600" MaxFileSize = "26214400" MaxRecursion = "16" MaxFiles = "1" MaxEmbeddedPE = "10485760" MaxHTMLNormalize = "10485760" MaxHTMLNoTags = "2097152" MaxScriptNormalize = "5242880" MaxZipTypeRcg = "1048576" MaxPartitions = "50" MaxIconsPE = "100" MaxRecHWP3 = "16" PCREMatchLimit = "1" PCRERecMatchLimit = "5000" PCREMaxFileSize = "26214400" ScanOnAccess = "yes" OnAccessMountPath disabled OnAccessIncludePath = "/tmp", "/home", "/root" OnAccessExcludePath disabled OnAccessExcludeUID disabled OnAccessMaxFileSize = "5242880" OnAccessDisableDDD disabled OnAccessPrevention = "yes" OnAccessExtraScanning disabled DevACOnly disabled DevACDepth disabled DevPerformance disabled DevLiblog disabled DisableCertCheck disabled Config file: freshclam.conf --- StatsHostID disabled StatsEnabled disabled StatsTimeout disabled LogFileMaxSize = "4294967295" LogTime = "yes" LogSyslog disabled LogFacility = "LOG_LOCAL6" LogVerbose disabled LogRotate = "yes" PidFile disabled DatabaseDirectory = "/var/lib/clamav" Foreground disabled Debug disabled UpdateLogFile = "/var/log/clamav/freshclam.log" DatabaseOwner = "clamav" Checks = "24" DNSDatabaseInfo = "current.cvd.clamav.net" DatabaseMirror = "db.local.clamav.net", "database.clamav.net" PrivateMirror disabled MaxAttempts = "5" ScriptedUpdates = "yes" TestDatabases = "yes" CompressLocalDatabase disabled
