2016-10-18 12:26 GMT+02:00 phrogz <phr...@phrogz.me>:
> Hello,
>
>
> 1/ I'm trying to understand why in impstats my action queue doesn't reach an 
> higher value than 1000 (and sometime there is a value for 'full', I don't 
> know why).

you have commented out "queue.size", so it uses the default of 1,000.
If you specify a larger size, that larger value will be used.

This is the same for DA as for other queues. What did you get the idea
it does not apply to DA?

Rainer
>
>
> e.g. :
>
>       "name": "main Q",
>       "origin": "core.queue",
>       "size": 10916,
>       "enqueued": 23215,
>       "full": 0,
>       "discarded_full": 0,
>       "discarded_nf": 0,
>       "maxqsize": 144059
>
>       "name": "action 1 queue",
>       "origin": "core.queue",
>       "size": 1000,
>       "enqueued": 17808,
>       "full": 63,
>       "discarded_full": 0,
>       "discarded_nf": 0,
>       "maxqsize": 1000
>
>       "name": "action 2 queue",
>       "origin": "core.queue",
>       "size": 0,
>       "enqueued": 0,
>       "full": 0,
>       "discarded_full": 0,
>       "discarded_nf": 0,
>       "maxqsize": 1000
>
>       "name": "action 1 queue[DA]",
>       "origin": "core.queue",
>       "size": 0,
>       "enqueued": 0,
>       "full": 0,
>       "discarded_full": 0,
>       "discarded_nf": 0,
>       "maxqsize": 0
>
>       "name": "action 2 queue[DA]",
>       "origin": "core.queue",
>       "size": 0,
>       "enqueued": 0,
>       "full": 0,
>       "discarded_full": 0,
>       "discarded_nf": 0,
>       "maxqsize": 0
>
>
> 2/ There is also something strange, there is two old queues in the rsyslog 
> spool that are not deleted.
> #####################
> #####################
> #####################
>
> here is the debug:
> 1768.304204149:main thread    : action 1 queue: starting queue
> 1768.304209377:main thread    : action 1 queue: is disk-assisted, disk will 
> be used on demand
> 1768.304216886:main thread    : action 1 queue: params: type 1, enq-only 0, 
> disk assisted 1, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, maxQSize 
> 1000, lqsize 0, pqsize 0, child 0, full delay 450, light delay 700, deq batch 
> size 1000, high wtrmrk 900, low wtrmrk 700, discardmrk 980, max wrkr 4, min 
> msgs f. wrkr 250
> 1768.304222849:main thread    : action 1 queue:Reg: finalizing construction 
> of worker thread pool (numworkerThreads 4)
> 1768.304228002:main thread    : action 1 queue:Reg/w0: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304236533:main thread    : action 1 queue:Reg/w1: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304243582:main thread    : action 1 queue:Reg/w2: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304260052:main thread    : action 1 queue:Reg/w3: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304275153:main thread    : action 1 queue:DAwpool: finalizing 
> construction of worker thread pool (numworkerThreads 1)
> 1768.304288963:main thread    : action 1 queue:DAwpool/w0: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304334755:main thread    : action 1 queue[DA]: starting queue
> 1768.304343275:main thread    : action 1 queue[DA]: .qi file name is 
> '/apps/rsyslog/spool/ToLogStash.qi', len 33
> 1768.304347249:main thread    : action 1 queue[DA]: I am a child
> 1768.304519153:main thread    : file stream /apps/rsyslog/spool/ToLogStash.qi 
> params: flush interval 0, async write 0
> 1768.304545844:main thread    : file '/apps/rsyslog/spool/ToLogStash.qi' 
> opened as #10 with mode 384
> 1768.304565987:main thread    : strm 0x7f43404b9890: opened file 
> '/apps/rsyslog/spool/ToLogStash.qi' for READ as 10
> 1768.304574600:main thread    : strm 0x7f43404b9890: file 10 read 543 bytes
> 1768.304603730:main thread    : file stream ToLogStash params: flush interval 
> 0, async write 0
> 1768.304626194:main thread    : file stream ToLogStash params: flush interval 
> 0, async write 0
> 1768.304636987:main thread    : file stream ToLogStash params: flush interval 
> 0, async write 0
> 1768.304658150:main thread    : file 
> '/apps/rsyslog/spool/ToLogStash.00038759' opened as #11 with mode 384
> 1768.304669771:main thread    : queue 'strm 0x7f43404bac20', file 
> '/apps/rsyslog/spool/ToLogStash.00038759' opened for non-append write, but 
> already contains 358469 bytes
> 1768.304674986:main thread    : strm 0x7f43404bac20: opened file 
> '/apps/rsyslog/spool/ToLogStash.00038759' for WRITE as 11
> 1768.304679648:main thread    : strm 0x7f43404bac20: file 11 seek, pos 639
> 1768.304691562:main thread    : file 
> '/apps/rsyslog/spool/ToLogStash.00038759' opened as #22 with mode 384
> 1768.304701188:main thread    : strm 0x7f43404bbeb0: opened file 
> '/apps/rsyslog/spool/ToLogStash.00038759' for READ as 22
> 1768.304705945:main thread    : strm 0x7f43404bbeb0: file 22 seek, pos 639
> 1768.304716357:main thread    : file 
> '/apps/rsyslog/spool/ToLogStash.00038759' opened as #23 with mode 384
> 1768.304726030:main thread    : strm 0x7f43404bd2a0: opened file 
> '/apps/rsyslog/spool/ToLogStash.00038759' for READ as 23
> 1768.304731297:main thread    : strm 0x7f43404bd2a0: file 23 seek, pos 639
> 1768.304736880:main thread    : strm 0x7f43404b9890: file 
> 10(/apps/rsyslog/spool/ToLogStash.qi) closing, bDeleteOnClose 0
> 1768.304750072:main thread    : action 1 queue[DA]: params: type 2, enq-only 
> 0, disk assisted 0, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, 
> maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1, light delay -1, deq 
> batch size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs 
> f. wrkr 0
> 1768.304758727:main thread    : action 1 queue[DA]:Reg: finalizing 
> construction of worker thread pool (numworkerThreads 1)
> 1768.304764348:main thread    : action 1 queue[DA]:Reg/w0: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304769386:main thread    : action 1 queue[DA]: queue finished 
> initialization
> 1768.304776843:main thread    : action 1 queue: DA queue initialized, disk 
> queue 0x7f43404b9550
> 1768.304781135:main thread    : action 1 queue: queue finished initialization
> 1768.304789476:main thread    : Action builtin:omfwd[0x7f4340483dc0]: queue 
> 0x7f4340484b60 started
> 1768.304794344:main thread    : iterateAllActions calling into action 
> 0x7f4340484ea0
> 1768.304798231:main thread    : action 2 queue: starting queue
> 1768.304803919:main thread    : action 2 queue: is disk-assisted, disk will 
> be used on demand
> 1768.304811645:main thread    : action 2 queue: params: type 1, enq-only 0, 
> disk assisted 1, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, maxQSize 
> 1000, lqsize 0, pqsize 0, child 0, full delay 450, light delay 700, deq batch 
> size 1000, high wtrmrk 900, low wtrmrk 700, discardmrk 980, max wrkr 4, min 
> msgs f. wrkr 250
> 1768.304817986:main thread    : action 2 queue:Reg: finalizing construction 
> of worker thread pool (numworkerThreads 4)
> 1768.304823043:main thread    : action 2 queue:Reg/w0: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304840673:main thread    : action 2 queue:Reg/w1: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304862659:main thread    : action 2 queue:Reg/w2: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304879227:main thread    : action 2 queue:Reg/w3: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304894375:main thread    : action 2 queue:DAwpool: finalizing 
> construction of worker thread pool (numworkerThreads 1)
> 1768.304899909:main thread    : action 2 queue:DAwpool/w0: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.304916529:main thread    : action 2 queue[DA]: starting queue
> 1768.304922466:main thread    : action 2 queue[DA]: .qi file name is 
> '/apps/rsyslog/spool/RelayToLogStash.qi', len 38
> 1768.304926530:main thread    : action 2 queue[DA]: I am a child
> 1768.304938902:main thread    : file stream 
> /apps/rsyslog/spool/RelayToLogStash.qi params: flush interval 0, async write 0
> 1768.304955158:main thread    : file '/apps/rsyslog/spool/RelayToLogStash.qi' 
> opened as #10 with mode 384
> 1768.304965285:main thread    : strm 0x7f43404cb7c0: opened file 
> '/apps/rsyslog/spool/RelayToLogStash.qi' for READ as 10
> 1768.304985578:main thread    : strm 0x7f43404cb7c0: file 10 read 560 bytes
> 1768.305009054:main thread    : file stream RelayToLogStash params: flush 
> interval 0, async write 0
> 1768.305064267:main thread    : file stream RelayToLogStash params: flush 
> interval 0, async write 0
> 1768.305074762:main thread    : file stream RelayToLogStash params: flush 
> interval 0, async write 0
> 1768.305151652:main thread    : file 
> '/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #24 with mode 384
> 1768.305165535:main thread    : queue 'strm 0x7f43404ccbe0', file 
> '/apps/rsyslog/spool/RelayToLogStash.00002857' opened for non-append write, 
> but already contains 1012893 bytes
> 1768.305170609:main thread    : strm 0x7f43404ccbe0: opened file 
> '/apps/rsyslog/spool/RelayToLogStash.00002857' for WRITE as 24
> 1768.305175510:main thread    : strm 0x7f43404ccbe0: file 24 seek, pos 438297
> 1768.305187379:main thread    : file 
> '/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #25 with mode 384
> 1768.305197024:main thread    : strm 0x7f43404cdfd0: opened file 
> '/apps/rsyslog/spool/RelayToLogStash.00002857' for READ as 25
> 1768.305202122:main thread    : strm 0x7f43404cdfd0: file 25 seek, pos 1817
> 1768.305212704:main thread    : file 
> '/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #26 with mode 384
> 1768.305221830:main thread    : strm 0x7f43404cf260: opened file 
> '/apps/rsyslog/spool/RelayToLogStash.00002857' for READ as 26
> 1768.305226641:main thread    : strm 0x7f43404cf260: file 26 seek, pos 1817
> 1768.305232314:main thread    : strm 0x7f43404cb7c0: file 
> 10(/apps/rsyslog/spool/RelayToLogStash.qi) closing, bDeleteOnClose 0
> 1768.305244857:main thread    : action 2 queue[DA]: params: type 2, enq-only 
> 0, disk assisted 0, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, 
> maxQSize 0, lqsize 731, pqsize 731, child 1, full delay -1, light delay -1, 
> deq batch size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min 
> msgs f. wrkr 0
> 1768.305253009:main thread    : action 2 queue[DA]:Reg: finalizing 
> construction of worker thread pool (numworkerThreads 1)
> 1768.305258858:main thread    : action 2 queue[DA]:Reg/w0: finalizing 
> construction of worker instance data (for 13 actions)
> 1768.305263853:main thread    : action 2 queue[DA]: queue finished 
> initialization
> 1768.305268754:main thread    : action 2 queue[DA]:Reg: high activity - 
> starting 1 additional worker thread(s).
> 1768.305374179:main thread    : action 2 queue[DA]:Reg: started with state 0, 
> num workers now 1
> 1768.305385824:main thread    : action 2 queue: DA queue initialized, disk 
> queue 0x7f43404cb3f0
> 1768.305390224:main thread    : action 2 queue: queue finished initialization
> 1768.305397556:main thread    : Action builtin:omfwd[0x7f4340484ea0]: queue 
> 0x7f4340485460 started
> 1768.305402568:main thread    : Activating Ruleset Queue[(nil)] for Ruleset 
> RSYSLOG_DefaultRuleset
> 1768.305406401:main thread    : Activating Ruleset Queue[(nil)] for Ruleset 
> local
> 1768.305417331:7f433317b700: thread created, tid 4945, name 'rs:action 2 
> queue[DA]:R'
> 1768.305498188:action 2 queue[DA]:Reg/w0: wti 0x7f43404cb8f0: worker starting
> 1768.305512959:main thread    : Activating Ruleset Queue[(nil)] for Ruleset 
> remote_tcp_secure_6514
> 1768.305536551:main thread    : Activating Ruleset Queue[(nil)] for Ruleset 
> remote_relay_secure_6514
> 1768.305542715:main thread    : activateMainQueue: mainq cnf obj ptr is 
> 0x7f434047ebb0
> 1768.305554811:main thread    : nvlstGetParam: name 'queue.size', type 9, 
> valnode->bUsed 0
> 1768.305560230:main thread    : nvlstGetParam: name 'queue.dequeuebatchsize', 
> type 6, valnode->bUsed 0
> 1768.305567499:main thread    : nvlstGetParam: name 'queue.workerthreads', 
> type 6, valnode->bUsed 0
> 1768.305573965:main thread    : queue param blk:
> 1768.305577887:main thread    : queue.filename: (unset)
> 1768.305594020:action 2 queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted 0 
> objects and enqueued 0 objects
> 1768.305616369:action 2 queue[DA]:Reg/w0: doDeleteBatch: delete batch from 
> store, new sizes: log 731, phys 731
> 1768.305630344:main thread    : queue.spooldirectory: (unset)
> 1768.305640736:action 2 queue[DA]:Reg/w0: strm 0x7f43404cf260: file 26 read 
> 4096 bytes
>
> 3/Here is my config:
> ############################
> ############################
> ############################
> global(
>   defaultNetstreamDriver="gtls"
>   #debug.gnutls="10"
>   #TCP TLS
>   defaultNetstreamDriverCAFile="/apps/rsyslog/certCA.crt"
>   defaultNetstreamDriverCertFile="/apps/rsyslog/certHost.crt"
>   defaultNetstreamDriverKeyFile="/apps/rsyslog/certHost.key"
> )
> module(load="omrelp")
> module(load="imptcp")
> module(
>   load="imtcp"
>   streamDriver.name="gtls"
>   streamDriver.mode="1"
>   streamDriver.authMode="x509/name"
>   permittedPeer=["sourcesrv1.test.lan","sourcesrv2.test.lan"]
> )
> module(
>  load="impstats"
>  interval="10"
>  resetCounters="on"
>  format="cee"
>  ruleset="remote_tcp_secure_6514"      # send stats to the 
> remote_tcp_secure_6514 ruleset
>  )
>
>
> template(name="SyslogSource" type="string"
>          string="<%pri%>%timestamp:::date-rfc3339% %hostname% %fromhost-ip% 
> %syslogtag% %msg%\n"
>         )
> template(name="SyslogSourceRelay" type="string"
>          string="<%pri%>%timestamp:::date-rfc3339% %hostname% %syslogtag% 
> %msg%\n"
>         )
>
> /* Input entries */
>
> input(type="imptcp" port="514" ruleset="remote_tcp_secure_6514")
> #input(type="imptcp" port="514" ruleset="remote_relp_secure_6514")
>
> input(
>   type="imtcp"
>   port="6514"
>   name="imtcp_secure"
>   ruleset="remote_relay_secure_6514"
> )
>
> main_queue(
>   queue.size="1000000"   # capacity of the main queue
>   queue.dequeuebatchsize="10000"  # process messages in batches of 1000 and 
> move them to the action queues
>   queue.workerthreads="2"  # 2 threads for the main queue
> )
>
> /* Remote Logging */
>
> RuleSet (name="remote_tcp_secure_6514")
> {
>     action(
>       type="omfwd"
>       target="destserv.test.lan"
>       port="6514"
>       protocol="tcp"
>       StreamDriverMode="1"
>       StreamDriver="gtls"
>       StreamDriverAuthMode="x509/name"
>       StreamDriverPermittedPeers="destserv.test.lan"
>       action.resumeRetryCount="-1"
>       queue.type="linkedList"
>       queue.filename="ToLogStash"
>       queue.spoolDirectory="/apps/rsyslog/spool"
>       queue.maxDiskSpace="298g"
>       queue.saveOnShutdown="on"
>       queue.dequeuebatchsize="2000"
>       RebindInterval="1000"
>       queue.workerthreads="4"
>       queue.highwatermark="500000"
>       queue.lowwatermark="200000"
>       template="SyslogSource"
>       #queue.size="1000000000" #Not needed for DA Queue
>       action.reportSuspension="on"
>       action.reportSuspensionContinuation="on"
>     )
> }
>
>
> RuleSet (name="remote_relay_secure_6514")
> {
>     action(
>       type="omfwd"
>       target="destserv.test.lan"
>       port="6514"
>       protocol="tcp"
>       StreamDriverMode="1"
>       StreamDriver="gtls"
>       StreamDriverAuthMode="x509/name"
>       StreamDriverPermittedPeers="destserv.test.lan"
>       action.resumeRetryCount="-1"
>       queue.type="linkedList"
>       queue.filename="RelayToLogStash"
>       queue.spoolDirectory="/apps/rsyslog/spool"
>       queue.maxDiskSpace="200g"
>       queue.saveOnShutdown="on"
>       queue.dequeuebatchsize="2000"
>       RebindInterval="1000"
>       queue.workerthreads="4"
>       queue.highwatermark="500000"
>       queue.lowwatermark="200000"
>       template="SyslogSourceRelay"
>       #queue.size="1000000000" #Not needed for DA Queue
>       action.reportSuspension="on"
>       action.reportSuspensionContinuation="on"
>     )
> }
>
> Thanks,
>
> Ludovic
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to