[rsyslog] Palo Alto Network device logging stopped working

[Tod A Sandman via rsyslog]

I am running a central log server with rsyslog-8.2006 and the latest release of 
RHEL7.
  
A few weeks ago logging stopped working for one type of client, our security 
group's (ISO) Palo Alto Network (PAN) devices.  These log to a dedicated port 
(9022) using SSL and imtcp.  I've tried both the ossl and gtls drivers and get 
the same result:  tcpdump shows network traffic from the clients and lsof shows 
established connections, but no logging is taking place.   Logging via the same 
port/config works fine for Linux rsyslog clients I've tested.  It sure seems 
like a client or a network issue to me:

  imtcp.c : nsd_ossl.c: osslRecordRecv: Errno 104, connection resetted by peer


But our ISO group is at a loss and I have no more ideas. I'm hoping that 
someone can look at my config and error logs and confirm this is a client issue 
and/or give me any suggestions for further debugging.

To minimize the configuration debugging and noise, today I had the ISO folk 
point a single PAN device to a standby rsyslog server identical to the 
production server.  I then removed all client logging configuration on the 
standby except for the PAN client configuration.

Attached (rsyslog_debug_fail.log.gz) is a debug log after starting rsyslog, 
waiting a few minutes, and stopping rsyslog.

Attached also are my config files.

Thanks for any advice.


Tod A. Sandman
Office of Information Technology
Rice University

rsyslog_debug_fail.log.gz
Description: application/gzip

##################################################################################
#
# /etc/rsyslog.conf - main rsyslog configuration file
#
##################################################################################


global (
  umask = "0000"
  workDirectory = "/var/lib/rsyslog"
  maxMessageSize = "8192"
  net.enableDNS = "off"
  parser.permitSlashInProgramName = "off"
  debug.logFile = "/var/log/rsyslog/debug"
  oversizemsg.errorfile = "/var/log/rsyslog/oversize"
  defaultNetstreamDriverCAFile = "/etc/pki/tls/certs/cacerts.pem"
  defaultNetstreamDriverCertFile = "/etc/pki/tls/certs/logs.crt"
  defaultNetstreamDriverKeyFile = "/etc/pki/tls/private/logs.key"
  debug.gnutls = "10"
  senders.keepTrack = "on"
  internalmsg.ratelimit.interval = "5"
  internalmsg.ratelimit.burst = "1000"
# internalmsg.severity = "info"
  internalmsg.severity = "debug"        ## Not in reccomented for production
)

#### MODULES ####

## Support for local system logging (e.g. via logger command):
#module (load="imuxsock" SysSock.Name="/run/systemd/journal/syslog")
#module (load="imuxsock")

## Provides access to the systemd journal:
#module (load="imjournal")

## Reads kernel messages (the same are read from journald):
#module (load="imklog")

## Provides --MARK-- message capability:
#module (load="immark" interval="3600")

## UDP syslog reception:
#module (load="imudp")

## Plain TCP syslog reception:
#module (load="imptcp")

## TCP syslog reception:
module (load="imtcp"
  MaxListeners = "30"
  MaxSessions = "1024"
  StreamDriver.Mode="1"
  StreamDriver.Name="ossl"
# StreamDriver.Name="gtls"
  StreamDriver.AuthMode="x509/name"
# StreamDriver.AuthMode="anon"
  PermittedPeer=["*.rice.edu","*.*.rice.edu","*.*.*.rice.edu","*.pphosted.com"]
)

## Output file support:
module (load="builtin:omfile"
 #Template="RSYSLOG_TraditionalFileFormat"
  Template="RSYSLOG_FileFormat"
  dirCreateMode="0755" dirOwner="root" dirGroup="logs"
  fileCreateMode="0640" fileOwner="root" fileGroup="logs"
)


#### Legacy GLOBAL DIRECTIVES ####

$MainMsgQueueSize  100000       # maximum size of the message queue
$MainMsgQueueWorkerThreads 100  # number of worker threads, default 1

## Allow each ruleset to use its own "main" message queue for message 
submission.
## This must be specifed within each Ruleset.
$RulesetCreateMainQueue on

# File syncing capability is disabled by default. This feature is usually not 
required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

## Increase message rate limit (default is 20000 per 600secs):
## Or disable rate limiting altogether (set to 0):
$imjournalRatelimitInterval 0
$imjournalRatelimitBurst 0

## Include all config files in /etc/rsyslog.d/
include (file="/etc/rsyslog.d/*.conf")

## Configure incoming logging from remote systems:
include (file="/etc/rsyslog.d/inputs/*.conf")
include (file="/etc/rsyslog.d/rules/*.conf")

## Define port 9022 listener, bound to the appropriate ruleset:

input(type="imtcp" name="tcp-9022" port="9022" ruleset="port9022") 

###############################################################################
#
# /etc/rsyslog.d/rules/port9022.conf - ruleset tied to the port 9022 input.
#
# ISO PAN configuration file
#
###############################################################################

template(name="iso-pan" type="list") {
  property(name="$.basedir")
  constant(value="/pan/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
}

template(name="port9022-catchall" type="list") {
  property(name="$.basedir")
  constant(value="/pan/catchall/")
  property(name="$year")
  property(name="$month")
  property(name="$day")
}


ruleset(name="port9022" queue.type="linkedList" queue.saveOnShutdown="on") {

  Set $.basedir = "/remote/logs/iso";

  ## ISO PAN servers:
  if (($fromhost != "168.7.56.225")) then {
    action(type="omfile" Template="RSYSLOG_FileFormat" DynaFile="iso-pan")
    stop
  }

  action(type="omfile" Template="RSYSLOG_FileFormat" 
DynaFile="port9022-catchall")
}


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.