ah, it's still a very interesting note. I think v8 does most probably have the same problem...
Rainer On Wed, Dec 4, 2013 at 11:48 AM, David Lang <[email protected]> wrote: > 5.8 is quite old at this point, there are a lot of changes in 8.1 that are > designed to significantly improve multi-threaded workloads, I would suggest > looking at that because there is no further development of the 5.x branch, > and things have probably changed in three major releases. > > David Lang > > On Wed, 4 Dec 2013, Grégoire Seux wrote: > > Date: Wed, 4 Dec 2013 10:41:29 +0000 >> From: Grégoire Seux <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: "rsyslog-users ([email protected])" < >> [email protected]> >> Subject: [rsyslog] regex instance shared between all workers >> >> >> Hello, >> >> I have noticed that all workers dequeing messages in main queue are using >> the same objects instances (at least in 5.8.10). >> This creates a lot of contention when using regexp since matching a >> string against a regexp take a lock on the regexp instance (in libc) >> >> As a result, all threads but one are waiting to acquire the lock. >> Here is a copy of one backtrace: >> >> Thread 4 (Thread 0x7f7ad75fe700 (LWP 25735)): >> #0 0x00007f7b532b306e in __lll_lock_wait_private () from /lib64/libc.so.6 >> #1 0x00007f7b5328852a in _L_lock_49262 () from /lib64/libc.so.6 >> #2 0x00007f7b532845fc in regexec@@GLIBC_2.3.4 () from /lib64/libc.so.6 >> #3 0x00007f7b541e4b80 in MsgGetProp (pMsg=<value optimized out>, >> pTpe=0x7f7b555fa500, >> propID=<value optimized out>, pPropLen=0x7f7ad75fd158, >> pbMustBeFreed=0x7f7ad75fd16e) >> at msg.c:2621 >> #4 0x00007f7b54204979 in tplToString (pTpl=0x7f7b555f9370, >> pMsg=0x7f7aa3ed1a40, >> ppBuf=0x7f7b519984a0, pLenBuf=0x7f7b519984c0) at ../template.c:129 >> #5 0x00007f7b54202305 in prepareDoActionParams (pAction=0x7f7b5560b370, >> pBatch=0x7f7b55641e48, pbShutdownImmediate=<value optimized out>) at >> ../action.c:780 >> #6 prepareBatch (pAction=0x7f7b5560b370, pBatch=0x7f7b55641e48, >> pbShutdownImmediate=<value optimized out>) at ../action.c:1166 >> #7 processBatchMain (pAction=0x7f7b5560b370, pBatch=0x7f7b55641e48, >> pbShutdownImmediate=<value optimized out>) at ../action.c:1207 >> #8 0x00007f7b541ff93f in doQueueEnqObjDirectBatch >> (pAction=0x7f7b5560b370, >> pBatch=0x7f7b55641e48) at ../action.c:1659 >> #9 0x00007f7b54200f2b in doSubmitToActionQNotAllMarkBatch >> (pAction=0x7f7b5560b370, >> pBatch=0x7f7b55641e48) at ../action.c:1568 >> #10 0x00007f7b541fbfd3 in processBatchDoActions (pData=0x7f7b5560b370, >> pParam=<value optimized out>) at rule.c:101 >> #11 0x00007f7b541e653f in llExecFunc (pThis=0x7f7b556090b0, >> pFunc=0x7f7b541fbfa0 <processBatchDoActions>, pParam=0x7f7b55641e48) >> at linkedlist.c:389 >> #12 0x00007f7b541fc3b3 in processBatch (pThis=0x7f7b55609060, >> pBatch=0x7f7b55641e48) >> at rule.c:292 >> #13 0x00007f7b541fb3de in processBatchDoRules (pData=0x7f7b55609060, >> pParam=0x7f7b55641e48) >> at ruleset.c:150 >> #14 0x00007f7b541e653f in llExecFunc (pThis=0x7f7b555f4410, >> pFunc=0x7f7b541fb3b0 <processBatchDoRules>, pParam=0x7f7b55641e48) at >> linkedlist.c:389 >> #15 0x00007f7b541fb866 in processBatch (pBatch=0x7f7b55641e48) at >> ruleset.c:232 >> #16 0x00007f7b541d3641 in msgConsumer (notNeeded=<value optimized out>, >> pBatch=0x7f7b55641e48, pbShutdownImmediate=0x7f7b55641028) at >> syslogd.c:719 >> #17 0x00007f7b541faedb in ConsumerReg (pThis=0x7f7b55641010, >> pWti=0x7f7b55641e20) >> at queue.c:1760 >> #18 0x00007f7b541f3e46 in wtiWorker (pThis=0x7f7b55641e20) at wti.c:313 >> #19 0x00007f7b541f393a in wtpWorker (arg=0x7f7b55641e20) at wtp.c:387 >> #20 0x00007f7b53b77851 in start_thread () from /lib64/libpthread.so.0 >> #21 0x00007f7b532a390d in clone () from /lib64/libc.so.6 >> >> >> This is sad because each worker could (should?) create its own instance >> of the regex. >> >> Do you agree with that conclusion? >> >> The configuration except is: >> $template All,"/var/opt/logs/technicals/%msg:R,ERE,1,ZERO:^ >> \+([a-zA-Z/0-9\-]+)\+(.*)--end%.log" >> $template RemoveCR, "%TIMESTAMP:::date-rfc3339% %HOSTNAME% >> %msg:R,ERE,2,FIELD:^ \+([a-zA-Z/0-9\-]+)\+(.*)--end%\n" >> :fromhost-ip,!isequal,"127.0.0.1" -?All;RemoveCR >> >> I can include the full configuration if needed. >> >> Right now the only solution I see would be to load balance messages >> between several actions doing exactly the same treatment (but maybe this >> would create new issues). >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
