Hello,

1/ I'm trying to understand why in impstats my action queue doesn't reach an 
higher value than 1000 (and sometime there is a value for 'full', I don't know 
why).


e.g. :

      "name": "main Q",
      "origin": "core.queue",
      "size": 10916,
      "enqueued": 23215,
      "full": 0,
      "discarded_full": 0,
      "discarded_nf": 0,
      "maxqsize": 144059

      "name": "action 1 queue",
      "origin": "core.queue",
      "size": 1000,
      "enqueued": 17808,
      "full": 63,
      "discarded_full": 0,
      "discarded_nf": 0,
      "maxqsize": 1000

      "name": "action 2 queue",
      "origin": "core.queue",
      "size": 0,
      "enqueued": 0,
      "full": 0,
      "discarded_full": 0,
      "discarded_nf": 0,
      "maxqsize": 1000

      "name": "action 1 queue[DA]",
      "origin": "core.queue",
      "size": 0,
      "enqueued": 0,
      "full": 0,
      "discarded_full": 0,
      "discarded_nf": 0,
      "maxqsize": 0

      "name": "action 2 queue[DA]",
      "origin": "core.queue",
      "size": 0,
      "enqueued": 0,
      "full": 0,
      "discarded_full": 0,
      "discarded_nf": 0,
      "maxqsize": 0


2/ There is also something strange, there is two old queues in the rsyslog 
spool that are not deleted.
#####################
#####################
#####################

here is the debug:
1768.304204149:main thread    : action 1 queue: starting queue
1768.304209377:main thread    : action 1 queue: is disk-assisted, disk will be 
used on demand
1768.304216886:main thread    : action 1 queue: params: type 1, enq-only 0, 
disk assisted 1, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, maxQSize 
1000, lqsize 0, pqsize 0, child 0, full delay 450, light delay 700, deq batch 
size 1000, high wtrmrk 900, low wtrmrk 700, discardmrk 980, max wrkr 4, min 
msgs f. wrkr 250
1768.304222849:main thread    : action 1 queue:Reg: finalizing construction of 
worker thread pool (numworkerThreads 4)
1768.304228002:main thread    : action 1 queue:Reg/w0: finalizing construction 
of worker instance data (for 13 actions)
1768.304236533:main thread    : action 1 queue:Reg/w1: finalizing construction 
of worker instance data (for 13 actions)
1768.304243582:main thread    : action 1 queue:Reg/w2: finalizing construction 
of worker instance data (for 13 actions)
1768.304260052:main thread    : action 1 queue:Reg/w3: finalizing construction 
of worker instance data (for 13 actions)
1768.304275153:main thread    : action 1 queue:DAwpool: finalizing construction 
of worker thread pool (numworkerThreads 1)
1768.304288963:main thread    : action 1 queue:DAwpool/w0: finalizing 
construction of worker instance data (for 13 actions)
1768.304334755:main thread    : action 1 queue[DA]: starting queue
1768.304343275:main thread    : action 1 queue[DA]: .qi file name is 
'/apps/rsyslog/spool/ToLogStash.qi', len 33
1768.304347249:main thread    : action 1 queue[DA]: I am a child
1768.304519153:main thread    : file stream /apps/rsyslog/spool/ToLogStash.qi 
params: flush interval 0, async write 0
1768.304545844:main thread    : file '/apps/rsyslog/spool/ToLogStash.qi' opened 
as #10 with mode 384
1768.304565987:main thread    : strm 0x7f43404b9890: opened file 
'/apps/rsyslog/spool/ToLogStash.qi' for READ as 10
1768.304574600:main thread    : strm 0x7f43404b9890: file 10 read 543 bytes
1768.304603730:main thread    : file stream ToLogStash params: flush interval 
0, async write 0
1768.304626194:main thread    : file stream ToLogStash params: flush interval 
0, async write 0
1768.304636987:main thread    : file stream ToLogStash params: flush interval 
0, async write 0
1768.304658150:main thread    : file '/apps/rsyslog/spool/ToLogStash.00038759' 
opened as #11 with mode 384
1768.304669771:main thread    : queue 'strm 0x7f43404bac20', file 
'/apps/rsyslog/spool/ToLogStash.00038759' opened for non-append write, but 
already contains 358469 bytes
1768.304674986:main thread    : strm 0x7f43404bac20: opened file 
'/apps/rsyslog/spool/ToLogStash.00038759' for WRITE as 11
1768.304679648:main thread    : strm 0x7f43404bac20: file 11 seek, pos 639
1768.304691562:main thread    : file '/apps/rsyslog/spool/ToLogStash.00038759' 
opened as #22 with mode 384
1768.304701188:main thread    : strm 0x7f43404bbeb0: opened file 
'/apps/rsyslog/spool/ToLogStash.00038759' for READ as 22
1768.304705945:main thread    : strm 0x7f43404bbeb0: file 22 seek, pos 639
1768.304716357:main thread    : file '/apps/rsyslog/spool/ToLogStash.00038759' 
opened as #23 with mode 384
1768.304726030:main thread    : strm 0x7f43404bd2a0: opened file 
'/apps/rsyslog/spool/ToLogStash.00038759' for READ as 23
1768.304731297:main thread    : strm 0x7f43404bd2a0: file 23 seek, pos 639
1768.304736880:main thread    : strm 0x7f43404b9890: file 
10(/apps/rsyslog/spool/ToLogStash.qi) closing, bDeleteOnClose 0
1768.304750072:main thread    : action 1 queue[DA]: params: type 2, enq-only 0, 
disk assisted 0, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, maxQSize 0, 
lqsize 0, pqsize 0, child 1, full delay -1, light delay -1, deq batch size 8, 
high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0
1768.304758727:main thread    : action 1 queue[DA]:Reg: finalizing construction 
of worker thread pool (numworkerThreads 1)
1768.304764348:main thread    : action 1 queue[DA]:Reg/w0: finalizing 
construction of worker instance data (for 13 actions)
1768.304769386:main thread    : action 1 queue[DA]: queue finished 
initialization
1768.304776843:main thread    : action 1 queue: DA queue initialized, disk 
queue 0x7f43404b9550
1768.304781135:main thread    : action 1 queue: queue finished initialization
1768.304789476:main thread    : Action builtin:omfwd[0x7f4340483dc0]: queue 
0x7f4340484b60 started
1768.304794344:main thread    : iterateAllActions calling into action 
0x7f4340484ea0
1768.304798231:main thread    : action 2 queue: starting queue
1768.304803919:main thread    : action 2 queue: is disk-assisted, disk will be 
used on demand
1768.304811645:main thread    : action 2 queue: params: type 1, enq-only 0, 
disk assisted 1, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, maxQSize 
1000, lqsize 0, pqsize 0, child 0, full delay 450, light delay 700, deq batch 
size 1000, high wtrmrk 900, low wtrmrk 700, discardmrk 980, max wrkr 4, min 
msgs f. wrkr 250
1768.304817986:main thread    : action 2 queue:Reg: finalizing construction of 
worker thread pool (numworkerThreads 4)
1768.304823043:main thread    : action 2 queue:Reg/w0: finalizing construction 
of worker instance data (for 13 actions)
1768.304840673:main thread    : action 2 queue:Reg/w1: finalizing construction 
of worker instance data (for 13 actions)
1768.304862659:main thread    : action 2 queue:Reg/w2: finalizing construction 
of worker instance data (for 13 actions)
1768.304879227:main thread    : action 2 queue:Reg/w3: finalizing construction 
of worker instance data (for 13 actions)
1768.304894375:main thread    : action 2 queue:DAwpool: finalizing construction 
of worker thread pool (numworkerThreads 1)
1768.304899909:main thread    : action 2 queue:DAwpool/w0: finalizing 
construction of worker instance data (for 13 actions)
1768.304916529:main thread    : action 2 queue[DA]: starting queue
1768.304922466:main thread    : action 2 queue[DA]: .qi file name is 
'/apps/rsyslog/spool/RelayToLogStash.qi', len 38
1768.304926530:main thread    : action 2 queue[DA]: I am a child
1768.304938902:main thread    : file stream 
/apps/rsyslog/spool/RelayToLogStash.qi params: flush interval 0, async write 0
1768.304955158:main thread    : file '/apps/rsyslog/spool/RelayToLogStash.qi' 
opened as #10 with mode 384
1768.304965285:main thread    : strm 0x7f43404cb7c0: opened file 
'/apps/rsyslog/spool/RelayToLogStash.qi' for READ as 10
1768.304985578:main thread    : strm 0x7f43404cb7c0: file 10 read 560 bytes
1768.305009054:main thread    : file stream RelayToLogStash params: flush 
interval 0, async write 0
1768.305064267:main thread    : file stream RelayToLogStash params: flush 
interval 0, async write 0
1768.305074762:main thread    : file stream RelayToLogStash params: flush 
interval 0, async write 0
1768.305151652:main thread    : file 
'/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #24 with mode 384
1768.305165535:main thread    : queue 'strm 0x7f43404ccbe0', file 
'/apps/rsyslog/spool/RelayToLogStash.00002857' opened for non-append write, but 
already contains 1012893 bytes
1768.305170609:main thread    : strm 0x7f43404ccbe0: opened file 
'/apps/rsyslog/spool/RelayToLogStash.00002857' for WRITE as 24
1768.305175510:main thread    : strm 0x7f43404ccbe0: file 24 seek, pos 438297
1768.305187379:main thread    : file 
'/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #25 with mode 384
1768.305197024:main thread    : strm 0x7f43404cdfd0: opened file 
'/apps/rsyslog/spool/RelayToLogStash.00002857' for READ as 25
1768.305202122:main thread    : strm 0x7f43404cdfd0: file 25 seek, pos 1817
1768.305212704:main thread    : file 
'/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #26 with mode 384
1768.305221830:main thread    : strm 0x7f43404cf260: opened file 
'/apps/rsyslog/spool/RelayToLogStash.00002857' for READ as 26
1768.305226641:main thread    : strm 0x7f43404cf260: file 26 seek, pos 1817
1768.305232314:main thread    : strm 0x7f43404cb7c0: file 
10(/apps/rsyslog/spool/RelayToLogStash.qi) closing, bDeleteOnClose 0
1768.305244857:main thread    : action 2 queue[DA]: params: type 2, enq-only 0, 
disk assisted 0, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, maxQSize 0, 
lqsize 731, pqsize 731, child 1, full delay -1, light delay -1, deq batch size 
8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0
1768.305253009:main thread    : action 2 queue[DA]:Reg: finalizing construction 
of worker thread pool (numworkerThreads 1)
1768.305258858:main thread    : action 2 queue[DA]:Reg/w0: finalizing 
construction of worker instance data (for 13 actions)
1768.305263853:main thread    : action 2 queue[DA]: queue finished 
initialization
1768.305268754:main thread    : action 2 queue[DA]:Reg: high activity - 
starting 1 additional worker thread(s).
1768.305374179:main thread    : action 2 queue[DA]:Reg: started with state 0, 
num workers now 1
1768.305385824:main thread    : action 2 queue: DA queue initialized, disk 
queue 0x7f43404cb3f0
1768.305390224:main thread    : action 2 queue: queue finished initialization
1768.305397556:main thread    : Action builtin:omfwd[0x7f4340484ea0]: queue 
0x7f4340485460 started
1768.305402568:main thread    : Activating Ruleset Queue[(nil)] for Ruleset 
RSYSLOG_DefaultRuleset
1768.305406401:main thread    : Activating Ruleset Queue[(nil)] for Ruleset 
local
1768.305417331:7f433317b700: thread created, tid 4945, name 'rs:action 2 
queue[DA]:R'
1768.305498188:action 2 queue[DA]:Reg/w0: wti 0x7f43404cb8f0: worker starting
1768.305512959:main thread    : Activating Ruleset Queue[(nil)] for Ruleset 
remote_tcp_secure_6514
1768.305536551:main thread    : Activating Ruleset Queue[(nil)] for Ruleset 
remote_relay_secure_6514
1768.305542715:main thread    : activateMainQueue: mainq cnf obj ptr is 
0x7f434047ebb0
1768.305554811:main thread    : nvlstGetParam: name 'queue.size', type 9, 
valnode->bUsed 0
1768.305560230:main thread    : nvlstGetParam: name 'queue.dequeuebatchsize', 
type 6, valnode->bUsed 0
1768.305567499:main thread    : nvlstGetParam: name 'queue.workerthreads', type 
6, valnode->bUsed 0
1768.305573965:main thread    : queue param blk:
1768.305577887:main thread    : queue.filename: (unset)
1768.305594020:action 2 queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted 0 
objects and enqueued 0 objects
1768.305616369:action 2 queue[DA]:Reg/w0: doDeleteBatch: delete batch from 
store, new sizes: log 731, phys 731
1768.305630344:main thread    : queue.spooldirectory: (unset)
1768.305640736:action 2 queue[DA]:Reg/w0: strm 0x7f43404cf260: file 26 read 
4096 bytes

3/Here is my config:
############################
############################
############################
global(
  defaultNetstreamDriver="gtls"
  #debug.gnutls="10"
  #TCP TLS
  defaultNetstreamDriverCAFile="/apps/rsyslog/certCA.crt"
  defaultNetstreamDriverCertFile="/apps/rsyslog/certHost.crt"
  defaultNetstreamDriverKeyFile="/apps/rsyslog/certHost.key"
)
module(load="omrelp")
module(load="imptcp")
module(
  load="imtcp"
  streamDriver.name="gtls"
  streamDriver.mode="1"
  streamDriver.authMode="x509/name"
  permittedPeer=["sourcesrv1.test.lan","sourcesrv2.test.lan"]
)
module(
 load="impstats"
 interval="10"
 resetCounters="on"
 format="cee"
 ruleset="remote_tcp_secure_6514"      # send stats to the 
remote_tcp_secure_6514 ruleset
 )


template(name="SyslogSource" type="string"
         string="<%pri%>%timestamp:::date-rfc3339% %hostname% %fromhost-ip% 
%syslogtag% %msg%\n"
        )
template(name="SyslogSourceRelay" type="string"
         string="<%pri%>%timestamp:::date-rfc3339% %hostname% %syslogtag% 
%msg%\n"
        )

/* Input entries */

input(type="imptcp" port="514" ruleset="remote_tcp_secure_6514")
#input(type="imptcp" port="514" ruleset="remote_relp_secure_6514")

input(
  type="imtcp"
  port="6514"
  name="imtcp_secure"
  ruleset="remote_relay_secure_6514"
)

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.dequeuebatchsize="10000"  # process messages in batches of 1000 and 
move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

/* Remote Logging */

RuleSet (name="remote_tcp_secure_6514")
{
    action(
      type="omfwd"
      target="destserv.test.lan"
      port="6514"
      protocol="tcp"
      StreamDriverMode="1"
      StreamDriver="gtls"
      StreamDriverAuthMode="x509/name"
      StreamDriverPermittedPeers="destserv.test.lan"
      action.resumeRetryCount="-1"
      queue.type="linkedList"
      queue.filename="ToLogStash"
      queue.spoolDirectory="/apps/rsyslog/spool"
      queue.maxDiskSpace="298g"
      queue.saveOnShutdown="on"
      queue.dequeuebatchsize="2000"
      RebindInterval="1000"
      queue.workerthreads="4"
      queue.highwatermark="500000"
      queue.lowwatermark="200000"
      template="SyslogSource"
      #queue.size="1000000000" #Not needed for DA Queue
      action.reportSuspension="on"
      action.reportSuspensionContinuation="on"
    )
}


RuleSet (name="remote_relay_secure_6514")
{
    action(
      type="omfwd"
      target="destserv.test.lan"
      port="6514"
      protocol="tcp"
      StreamDriverMode="1"
      StreamDriver="gtls"
      StreamDriverAuthMode="x509/name"
      StreamDriverPermittedPeers="destserv.test.lan"
      action.resumeRetryCount="-1"
      queue.type="linkedList"
      queue.filename="RelayToLogStash"
      queue.spoolDirectory="/apps/rsyslog/spool"
      queue.maxDiskSpace="200g"
      queue.saveOnShutdown="on"
      queue.dequeuebatchsize="2000"
      RebindInterval="1000"
      queue.workerthreads="4"
      queue.highwatermark="500000"
      queue.lowwatermark="200000"
      template="SyslogSourceRelay"
      #queue.size="1000000000" #Not needed for DA Queue
      action.reportSuspension="on"
      action.reportSuspensionContinuation="on"
    )
}

Thanks,

Ludovic


_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to