Allow me to ask a seemingly dumb question:
Did you reboot after restoring the snapshot?
And another:
How many processes do you have reading from snmptrapd.log?
I.e.:
# lsof /usr/local/pf/logs/snmptrapd.log
This looks to me like a case of multiple (stale) processes trying to read and
write from the same resources.
Also check the output of ‘ps’ for multiple instances of pfmon and
pfdhcplistener.
Regards,
--
Louis Munro
[email protected] :: www.inverse.ca
+1.514.447.4918 x125 :: +1 (866) 353-6153 x125
Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence
(www.packetfence.org)
On Apr 24, 2015, at 13:20 , Muller, Daniel (HEALTH)
<[email protected]> wrote:
> Thank you, I have two types of VMware snapshots available, machine state with
> and without RAM.
>
> Rolling back the snapshot is normally expected to revert the entire system
> state undoing all changes including filesystem, database, etc.
>
> The snapshot essentially enables a copy on write shadow file for the virtual
> disk, reverting discards this.
> "Snapshot with memory" mode creates a second shadow file representing memory
> changes.
>
> I'm including my pf.conf, networks.conf and ip s output, along with an innodb
> status report:
> ( passwords and redundant lines removed ..)
> ---
> [root@pkfn pf]# ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
> 1000
> link/ether 00:50:56:a5:98:cf brd ff:ff:ff:ff:ff:ff
> inet6 fe80::250:56ff:fea5:98cf/64 scope link
> valid_lft forever preferred_lft forever
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
> 1000
> link/ether 00:50:56:a5:c2:1a brd ff:ff:ff:ff:ff:ff
> inet 10.50.156.54/23 brd 10.50.157.255 scope global eth1
> inet6 fe80::250:56ff:fea5:c21a/64 scope link
> valid_lft forever preferred_lft forever
> 4: eth0.110@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UP
> link/ether 00:50:56:a5:98:cf brd ff:ff:ff:ff:ff:ff
> inet 10.67.248.2/24 brd 10.67.248.255 scope global eth0.110
> inet6 fe80::250:56ff:fea5:98cf/64 scope link
> valid_lft forever preferred_lft forever
> 5: eth0.111@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UP
> link/ether 00:50:56:a5:98:cf brd ff:ff:ff:ff:ff:ff
> inet 10.67.252.2/24 brd 10.67.252.255 scope global eth0.111
> inet6 fe80::250:56ff:fea5:98cf/64 scope link
> valid_lft forever preferred_lft forever
> [root@pkfn pf]# cat conf/pf.conf |sed 's/pass\=.*$/pass=top_secret/'
> [general]
> #
> # general.domain
> #
> # Domain name of PacketFence system.
> domain=wadsworth.org
> #
> # general.hostname
> #
> # Hostname of PacketFence system. This is concatenated with the domain in
> Apache rewriting rules and therefore must be resolvable by clients.
> hostname=pkfn
> #
> # general.dnsservers
> #
> # Comma-delimited list of DNS servers. Passthroughs are created to allow
> queries to these servers from even "trapped" nodes.
> dnsservers=199.184.30.42,199.184.28.129
> #
> # general.dhcpservers
> #
> # Comma-delimited list of DHCP servers. Passthroughs are created to allow
> DHCP transactions from even "trapped" nodes.
> dhcpservers=199.184.30.27,10.50.156.21,10.49.66.7,10.57.48.14,199.184.28.130
>
> #
> # general.maintenance_interval
> #
> # Interval at which Packetfence runs its maintenance tasks.
> #maintenance_interval=1200s
> [trapping]
> #
> # trapping.range
> #
> # Comma-delimited list of address ranges/CIDR blocks that PacketFence will
> monitor/detect/trap on. Gateway, network, and
> # broadcast addresses are ignored.
> range=10.67.248.0/24,10.67.252.0/24
> #
> # trapping.passthrough
> #
> # When enabled, pfdns will resolve the real IP addresses of passthroughs and
> add them in the ipset session to give access
> # to trapped devices. Don´t forget to enable ip_forward on your server.
> passthrough=enabled
> #
> # trapping.proxy_passthroughs
> #
> # Comma-delimited list of domains to be use for apache passthrough
> proxy_passthroughs=inside.wadsworth.org,admin.wadsworth.org,loki.wadsworth.org,info.wadsworth.org
> #
> # trapping.interception_proxy
> #
> # When enabled, packetfence will intercept proxy request to somes specified
> port
> interception_proxy=enabled
> #
> # trapping.interception_proxy_port
> #
> # Comma-delimited list of port used to intercept proxy traffic
> interception_proxy_port=3128,8080
>
> #
> # registration.range
> #
> #
> #range=10.67.248.0/24
> [guests_admin_registration]
> #
> # guests_admin_registration.access_duration_choices
> #
> # These are all the choices offered in the guest management interface as
> # possible access duration values for a given registration.
> access_duration_choices=1DF+0D,1WF+0D,3MF+0D,1YF+1D,3YF
> #
> # guests_admin_registration.default_access_duration
> #
> # This is the default access duration value selected in the dropdown on the
> # guest management interface.
> default_access_duration=3MF+0D
>
> [alerting]
> #
> # alerting.emailaddr
> #
> # Email address to which notifications of rogue DHCP servers, violations with
> an action of "email", or any other
> # PacketFence-related message goes to.
> [email protected]
> #
> # alerting.smtpserver
> #
> # Server through which to send messages to the above emailaddr. The default
> is localhost - be sure you're running an SMTP
> # host locally if you don't change it!
> smtpserver=smtp.wadsworth.org
> #
> # alerting.subjectprefix
> #
> #Subject prefix for email notifications of rogue DHCP servers, violations
> with an action of "email", or any other
> #PacketFence-related message.
> subjectprefix=PKFN ALERT:
>
> [scan]
> #
> # scan.engine
> #
> # Which scan engine to use to perform client-side policy compliance.
> engine=nessus
> #
> # scan.duration
> #
> # Approximate duration of a scan. User being scanned on registration are
> presented a progress bar
> # for this duration, afterwards the browser refreshes until scan is complete.
> duration=10m
> #
> # scan.host
> #
> # Host the scanning engine is running on. For performance reasons, we
> # recommend running the scanning engine on a remote server. A passthrough will
> # be automagically created.
> host=vnix.wadsworth.org
> #
> # scan.user
> #
> # Username to log into scanning engine with.
> user=pkfn
> #
> # scan.pass
> #
> # Password to log into scanning engine with.
> pass=top_secret
> #
> # scan.nessus_clientpolicy
> #
> # Name of the remote policy on the nessus server
> nessus_clientpolicy=basic-policy
>
> [database]
> #
> # database.pass
> #
> # Password for the mysql database used by PacketFence.
> pass=top_secret
> #
> # database.user
> #
> # Username of the account with access to the mysql database used by
> PacketFence.
> user=pkfn
>
> [expire]
> #
> # expire.node
> #
> # Time before a node is removed due to inactivity.
> # A value of 0D disables expiration.
> # example:
> # node=90D
> node=120D
> #
> # expire.iplog
> #
> # Time which you would like to keep logs on IP/MAC information.
> # A value of 0D disables expiration.
> # example:
> # iplog=180D
> iplog=120D
> #
> # expire.traplog
> #
> # Time which you would like to keep logs on trap information.
> # A value of 0D disables expiration.
> # example:
> # traplog=180D
> traplog=60D
> #
> # expire.locationlog
> #
> # Time which you would like to keep logs on location information
> # Please note that this table should not become too big since it
> # could degrade pfsetvlan performance.
> # A value of 0D disables expiration.
> # example:
> # locationlog=180D
> locationlog=60D
>
> [services]
> #
> # services.iptables
> #
> # Should iptables be managed by PacketFence?
> iptables=disabled
>
> [captive_portal]
> #
> # captive_portal.network_detection_ip
> #
> # This IP is used as the webserver who hosts the
> common/network-access-detection.gif which is used to detect if network
> # access was enabled.
> # It cannot be a domain name since it is used in registration or quarantine
> where DNS is blackholed.
> # It is recommended that you allow your users to reach your packetfence
> server and put your LAN's PacketFence IP.
> # By default we will make this reach PacketFence's website as an easy
> solution.
> #
> network_detection_ip=199.184.30.46
> #
> # captive_portal.secure_redirect
> #
> # If secure_redirect is enabled, the captive portal uses HTTPS when
> redirecting
> # captured clients. This is the default behavior.
> secure_redirect=disabled
>
> [interface eth0.111]
> enforcement=vlan
> ip=10.67.252.2
> type=internal
> mask=255.255.255.0
>
> [interface eth0.110]
> enforcement=vlan
> ip=10.67.248.2
> type=internal
> mask=255.255.255.0
>
> [interface eth1]
> ip=10.50.156.54
> type=management
> mask=255.255.254.0
> [root@pkfn pf]# cat conf/networks.conf
> [10.67.252.0]
> dns=10.67.252.2
> dhcp_start=10.67.252.10
> gateway=10.67.252.2
> domain-name=pkfn-isolation.wadsworth.org
> named=enabled
> dhcp_max_lease_time=3600
> dhcpd=enabled
> type=vlan-isolation
> netmask=255.255.255.0
> dhcp_end=10.67.252.246
> dhcp_default_lease_time=300
>
> [10.67.248.0]
> dns=10.67.248.2
> dhcp_start=10.67.248.10
> gateway=10.67.248.2
> domain-name=pkfn-registration.wadsworth.org
> named=enabled
> dhcp_max_lease_time=3600
> dhcpd=enabled
> type=vlan-registration
> netmask=255.255.255.0
> dhcp_end=10.67.248.254
> dhcp_default_lease_time=300
>
> [10.67.249.0]
> dns=10.67.248.2
> next_hop=10.67.248.1
> gateway=10.67.249.1
> dhcp_start=10.67.249.10
> domain-name=vlan-registration.wadsworth.org
> named=enabled
> dhcp_max_lease_time=3600
> dhcpd=enabled
> netmask=255.255.255.0
> type=vlan-registration
> dhcp_end=10.67.249.254
> dhcp_default_lease_time=300
>
> [10.67.253.0]
> dns=10.67.252.2
> next_hop=10.67.252.1
> gateway=10.67.253.1
> dhcp_start=10.67.253.10
> domain-name=vlan-isolation.wadsworth.org
> named=enabled
> dhcp_max_lease_time=3600
> dhcpd=enabled
> netmask=255.255.255.0
> type=vlan-isolation
> dhcp_end=10.67.253.254
> dhcp_default_lease_time=300
>
> [10.67.250.0]
> dns=10.67.248.2
> next_hop=10.67.248.1
> gateway=10.67.250.1
> dhcp_start=10.67.250.10
> domain-name=vlan-registration.wadsworth.org
> named=enabled
> dhcp_max_lease_time=3600
> dhcpd=enabled
> netmask=255.255.255.0
> type=vlan-registration
> dhcp_end=10.67.250.254
> dhcp_default_lease_time=300
>
> [10.67.254.0]
> dns=10.67.252.2
> next_hop=10.67.252.1
> gateway=10.67.254.1
> dhcp_start=10.67.254.10
> domain-name=vlan-isolation.wadsworth.org
> named=enabled
> dhcp_max_lease_time=3600
> dhcpd=enabled
> netmask=255.255.255.0
> type=vlan-isolation
> dhcp_end=10.67.254.254
> dhcp_default_lease_time=300
>
> [10.67.251.0]
> dns=10.67.248.2
> next_hop=10.67.248.1
> gateway=10.67.251.1
> dhcp_start=10.67.251.10
> domain-name=vlan-registration.wadsworth.org
> named=enabled
> dhcp_max_lease_time=3600
> dhcpd=enabled
> netmask=255.255.255.0
> type=vlan-registration
> dhcp_end=10.67.251.254
> dhcp_default_lease_time=300
>
> [10.67.255.0]
> dns=10.67.252.2
> next_hop=10.67.252.1
> gateway=10.67.255.1
> dhcp_start=10.67.255.10
> domain-name=vlan-isolation.wadsworth.org
> named=enabled
> dhcp_max_lease_time=3600
> dhcpd=enabled
> netmask=255.255.255.0
> type=vlan-isolation
> dhcp_end=10.67.255.254
> dhcp_default_lease_time=300
>
>
> --- MySQL stuff :
> mysql> show innodb status;
>
> (removed several 100 boring lines like ------ )
>
> | InnoDB | |
> =====================================
> 150424 13:13:29 INNODB MONITOR OUTPUT
> =====================================
> Per second averages calculated from the last 33 seconds
> ----------
> SEMAPHORES
> ----------
> OS WAIT ARRAY INFO: reservation count 3429, signal count 3338
> Mutex spin waits 0, rounds 48601, OS waits 1745
> RW-shared spins 3544, OS waits 1655; RW-excl spins 19, OS waits 14
> ------------------------
> LATEST DETECTED DEADLOCK
> ------------------------
> 150424 13:10:15
> *** (1) TRANSACTION:
> TRANSACTION 0 167780540, ACTIVE 1 sec, process no 3540, OS thread id
> 139700189165312 fetching rows
> mysql tables in use 1, locked 1
> LOCK WAIT 5 lock struct(s), heap size 1216, 5 row lock(s)
> MySQL thread id 15, query id 14222 localhost pkfn Searching rows for update
> update iplog set end_time=now() where mac='00:03:93:7c:ba:a2' and (end_time=0
> or end_time > now())
> *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
> RECORD LOCKS space id 0 page no 1182 n bits 272 index `GEN_CLUST_INDEX` of
> table `pf`.`iplog` trx id 0 167780540 lock_mode X locks rec but not gap
> waiting
> Record lock, heap no 112
> *** (2) TRANSACTION:
> TRANSACTION 0 167780536, ACTIVE 3 sec, process no 3540, OS thread id
> 139700187035392 fetching rows, thread declared inside InnoDB 96
> mysql tables in use 1, locked 1
> 8675 lock struct(s), heap size 980976, 1825200 row lock(s), undo log entries 2
> MySQL thread id 23, query id 14208 localhost pkfn updating
> delete from iplog where unix_timestamp(end_time) < (unix_timestamp(now()) -
> '10368000') and end_time!=0
> *** (2) HOLDS THE LOCK(S):
> RECORD LOCKS space id 0 page no 1182 n bits 272 index `GEN_CLUST_INDEX` of
> table `pf`.`iplog` trx id 0 167780536 lock_mode X
> Record lock, heap no 1
>
> (... removed sequence of 1..203 ..)
>
> Record lock, heap no 203
> *** (2) WAITING FOR THIS LOCK TO BE GRANTED:
> RECORD LOCKS space id 0 page no 36673 n bits 280 index `GEN_CLUST_INDEX` of
> table `pf`.`iplog` trx id 0 167780536 lock_mode X waiting
> Record lock, heap no 167 PHYSICAL RECORD: n_fields 7; compact format; info
> bits 0
> 0: len 6; hex 0000002c1e7f; asc , ;; 1: len 6; hex 00000a0016c1; asc
> ;; 2: len 7; hex 80000004ec0110; asc ;; 3: len 17; hex
> 30303a30333a39333a37633a62613a6132; asc 00:03:93:7c:ba:a2;; 4: len 12; hex
> 31302e36332e31392e323030; asc 10.63.19.200;; 5: len 8; hex 80001253a2df33bf;
> asc S 3 ;; 6: len 8; hex 80001253b4b1947f; asc S ;;
>
> *** WE ROLL BACK TRANSACTION (1)
> ------------
> TRANSACTIONS
> ------------
> Trx id counter 0 167780676
> Purge done for trx's n:o < 0 167780675 undo n:o < 0 0
> History list length 28
> LIST OF TRANSACTIONS FOR EACH SESSION:
> ---TRANSACTION 0 0, not started, process no 3540, OS thread id 139700054947584
> MySQL thread id 39, query id 14698 localhost root
> show innodb status
> ---TRANSACTION 0 167778539, not started, process no 3540, OS thread id
> 139700055213824
> MySQL thread id 37, query id 8984 localhost pkfn
> ---TRANSACTION 0 167778481, not started, process no 3540, OS thread id
> 139700055480064
> MySQL thread id 36, query id 8838 localhost pkfn
> ---TRANSACTION 0 167778396, not started, process no 3540, OS thread id
> 139700055746304
> MySQL thread id 35, query id 9959 localhost pkfn
> ---TRANSACTION 0 167778862, not started, process no 3540, OS thread id
> 139700056012544
> MySQL thread id 34, query id 9893 localhost pkfn
> ---TRANSACTION 0 167778810, not started, process no 3540, OS thread id
> 139700056278784
> MySQL thread id 33, query id 9758 localhost pkfn
> ---TRANSACTION 0 167778717, not started, process no 3540, OS thread id
> 139700056545024
> MySQL thread id 32, query id 9553 localhost pkfn
> ---TRANSACTION 0 0, not started, process no 3540, OS thread id 139700056811264
> MySQL thread id 31, query id 9311 localhost pkfn
> ---TRANSACTION 0 167778590, not started, process no 3540, OS thread id
> 139700057077504
> MySQL thread id 30, query id 9185 localhost pkfn
> ---TRANSACTION 0 167778099, not started, process no 3540, OS thread id
> 139700057343744
> MySQL thread id 29, query id 9068 localhost pkfn
> ---TRANSACTION 0 167780492, not started, process no 3540, OS thread id
> 139700057609984
> MySQL thread id 28, query id 14314 localhost pkfn
> ---TRANSACTION 0 167780667, not started, process no 3540, OS thread id
> 139700057876224
> MySQL thread id 27, query id 14670 localhost pkfn
> ---TRANSACTION 0 167780665, not started, process no 3540, OS thread id
> 139700058408704
> MySQL thread id 25, query id 14666 localhost pkfn
> ---TRANSACTION 0 167780668, not started, process no 3540, OS thread id
> 139700058142464
> MySQL thread id 26, query id 14671 localhost pkfn
> ---TRANSACTION 0 167780666, not started, process no 3540, OS thread id
> 139700058674944
> MySQL thread id 24, query id 14669 localhost pkfn
> ---TRANSACTION 0 167780671, not started, process no 3540, OS thread id
> 139700187567872
> MySQL thread id 21, query id 14690 localhost pkfn
> ---TRANSACTION 0 167780673, not started, process no 3540, OS thread id
> 139700187035392
> MySQL thread id 23, query id 14694 localhost pkfn
> ---TRANSACTION 0 167775146, not started, process no 3540, OS thread id
> 139700187301632
> MySQL thread id 22, query id 14668 localhost pkfn
> ---TRANSACTION 0 167780483, not started, process no 3540, OS thread id
> 139700187834112
> MySQL thread id 20, query id 14298 localhost pkfn
> ---TRANSACTION 0 167780388, not started, process no 3540, OS thread id
> 139700188100352
> MySQL thread id 19, query id 14315 localhost pkfn
> ---TRANSACTION 0 167780579, not started, process no 3540, OS thread id
> 139700188366592
> MySQL thread id 18, query id 14296 localhost pkfn
> ---TRANSACTION 0 167780513, not started, process no 3540, OS thread id
> 139700188632832
> MySQL thread id 17, query id 14299 localhost pkfn
> ---TRANSACTION 0 167780627, not started, process no 3540, OS thread id
> 139700189165312
> MySQL thread id 15, query id 14474 localhost pkfn
> ---TRANSACTION 0 167780498, not started, process no 3540, OS thread id
> 139700188899072
> MySQL thread id 16, query id 14476 localhost pkfn
> ---TRANSACTION 0 167780445, not started, process no 3540, OS thread id
> 139700189431552
> MySQL thread id 14, query id 14477 localhost pkfn
> ---TRANSACTION 0 167780570, not started, process no 3540, OS thread id
> 139700189697792
> MySQL thread id 13, query id 14283 localhost pkfn
> ---TRANSACTION 0 167780585, not started, process no 3540, OS thread id
> 139700189964032
> MySQL thread id 12, query id 14311 localhost pkfn
> ---TRANSACTION 0 167775078, not started, process no 3540, OS thread id
> 139700191827712
> MySQL thread id 11, query id 788 localhost pkfn
> ---TRANSACTION 0 167775074, not started, process no 3540, OS thread id
> 139700190230272
> MySQL thread id 10, query id 782 localhost pkfn
> ---TRANSACTION 0 167780675, not started, process no 3540, OS thread id
> 139700191561472
> MySQL thread id 5, query id 14697 localhost pkfn
> --------
> FILE I/O
> --------
> I/O thread 0 state: waiting for i/o request (insert buffer thread)
> I/O thread 1 state: waiting for i/o request (log thread)
> I/O thread 2 state: waiting for i/o request (read thread)
> I/O thread 3 state: waiting for i/o request (write thread)
> Pending normal aio reads: 0, aio writes: 0,
> ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
> Pending flushes (fsync) log: 0; buffer pool: 0
> 101632 OS file reads, 4712 OS file writes, 2299 OS fsyncs
> 28.57 reads/s, 190144 avg bytes/read, 0.45 writes/s, 0.15 fsyncs/s
> -------------------------------------
> INSERT BUFFER AND ADAPTIVE HASH INDEX
> -------------------------------------
> Ibuf: size 1, free list len 0, seg size 2,
> 71 inserts, 71 merged recs, 60 merges
> Hash table size 17393, node heap has 1 buffer(s)
> 6.30 hash searches/s, 8.36 non-hash searches/s
> ---
> LOG
> ---
> Log sequence number 1 3254220
> Log flushed up to 1 3254220
> Last checkpoint at 1 3254220
> 0 pending log writes, 0 pending chkp writes
> 1636 log i/o's done, 0.09 log i/o's/second
> ----------------------
> BUFFER POOL AND MEMORY
> ----------------------
> Total memory allocated 20819512; in additional pool allocated 1048576
> Dictionary memory allocated 260680
> Buffer pool size 512
> Free buffers 43
> Database pages 468
> Modified db pages 0
> Pending reads 0
> Pending writes: LRU 0, flush list 0, single page 0
> Pages read 1127268, created 34, written 2910
> 331.63 reads/s, 0.00 creates/s, 0.39 writes/s
> Buffer pool hit rate 995 / 1000
> --------------
> ROW OPERATIONS
> --------------
> 0 queries inside InnoDB, 0 queries in queue
> 1 read views open inside InnoDB
> Main thread process no. 3540, id 139699974825728, state: waiting for server
> activity
> Number of rows inserted 586, updated 701, deleted 514, read 217214852
> 0.00 inserts/s, 0.00 updates/s, 0.03 deletes/s, 64374.96 reads/s
> ----------------------------
> END OF INNODB MONITOR OUTPUT
> ============================
> |
> +--------+------( ... removed more boring lines ... ----+ )
> 1 row in set, 1 warning (0.00 sec)
>
> mysql> \q
>
>
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
PacketFence-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/packetfence-users
