Either I'm missing something obvious, or there is a regression between 7.4
and 7.6 in how disk assisted queues operate under load.
After upgrading from the Debian package 7.4.8-1~bpo70+1 to
7.6.3-2~bpo70+1, I noticed that while data is spooled to disk when a
destination is backlogged, the data is never reloaded from the disk queue.
Stats output:
rsyslogd-pstats:action 7 queue[DA]: size=165256 enqueued=173248 full=0
discarded.full=0 discarded.nf=0 maxqsize=165488
rsyslogd-pstats:action 7 queue: size=1023 enqueued=100094084 full=0
discarded.full=0 discarded.nf=0 maxqsize=57840
Action configuration:
$WorkDirectory /var/spool/rsyslog # default location for work (spool)
files
$ActionQueueType FixedArray # Preallocate memory for the queue
$ActionQueueFileName logstash # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueCheckpointInterval 10000 # Checkpoint once every 10000 records
$ActionQueueSize 64000
$ActionQueueHighWaterMark 57600 # Start spooling to disk
$ActionQueueLowWaterMark 44800 # Stop spooling to disk
$ActionQueuediscardMark 63000 # Start throwing away data
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionQueueDequeueBatchSize 1000
if $inputname != "imtcp" then {
*.* @@rsyslog1:6514;ls_json
}
Since I have the low water mark set to 44800, and the current in-memory
queue size was 1023, I would expect to see data being loaded from the
queue, but the size of the on-disk queue stays steady.
Queue info file:
<OPB:1:qqueue:1:
+iQueueSize:2:6:846096:
+tVars.disk.sizeOnDisk:2:9:706319606:
>End
.
<Obj:1:strm:1:
+iCurrFNum:2:3:825:
+pszFName:1:8:logstash:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:0:
+sType:2:1:1:
+tOperationsMode:2:1:2:
+tOpenMode:2:3:384:
+iCurrOffs:2:6:281576:
+inode:2:1:0:
>End
.
<Obj:1:strm:1:
+iCurrFNum:2:3:821:
+pszFName:1:8:logstash:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:1:
+sType:2:1:1:
+tOperationsMode:2:1:1:
+tOpenMode:2:3:384:
+iCurrOffs:2:6:754561:
+inode:2:1:0:
>End
.
If I understand the queue info file, then data files 821-825 should be in
use, but there are currently 683 data files in the directory, with serial
numbers from 2 through 833, with a number of gaps in the sequence.
I didn't see this behavior under 7.4; did I make a stupid mistake with the
config, or is something else going on?
-Jeff
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
