Re: [ossec-list] Re: syscheckd causing soft lockups

2017-03-23 Thread John Gelnaw 

Upgrading has not solved the problem.

Still appears to be some form of port / bind issue based on the backtrace. 
 To obfuscate things, this was my ossec master (wazuh docker image), so it 
was running in a docker container, on a virtual machine under VMWare.

Nothing complicated there, right?

I'd love to hear any suggestions on where to look next to track down this 
problem.  I can (apparently) get around it by disabling rootcheck, but 
since that's one of the key features of ossec I really want for security, 
it's not a very good solution.



NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! 
[ossec-syscheckd:16223]
Modules linked in: xt_nat veth binfmt_misc ipt_MASQUERADE 
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 
nf_nat_ipv4 xt_addrtyp
e xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc 
iptable_filter vmw_vsock_vmci_transport vsock btrfs zlib_deflate raid6_pq 
xor intel_p
owerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev 
aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd pcspkr 
sg vmw
_vmci i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd 
grace sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi
 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common 
crc32c_intel vmwgfx drm_kms_helper ata_piix syscopyarea serio_raw 
sysfillrect
 sysimgblt fb_sys_fops ttm vmxnet3 drm libata vmw_pvscsi
 i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 16223 Comm: ossec-syscheckd Not tainted 
3.10.0-514.10.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference 
Platform, BIOS 6.00 09/21/2015
task: 88000593ce70 ti: 8800130ec000 task.ti: 8800130ec000
RIP: 0010:[]  [] 
_raw_spin_lock+0x32/0x50
RSP: 0018:8800130efde0  EFLAGS: 0203
RAX: 411c RBX: 0020 RCX: bb00
RDX: 384c RSI: 384c RDI: c900016fe4f0
RBP: 8800130efde0 R08: 8800b7aa9380 R09: c900016fe4f0
R10: 0008 R11: 0206 R12: c900016fe3e0
R13: 88013ae99a80 R14: 0246 R15: 8800130efd78
FS:  7efe439a5740() GS:88013ae8() knlGS:
CS:  0010 DS:  ES:  CR0: 8005003b
CR2: 0063e000 CR3: 13e8c000 CR4: 000407e0
DR0:  DR1:  DR2: 
DR3:  DR6: 0ff0 DR7: 0400
Stack:
 8800130efe68 815bc2b5 0005811de175 8800b8993640
    81f96140
  c900016fe4f0 0c01 
Call Trace:
 [] inet_csk_get_port+0x385/0x5c0
 [] inet_bind+0x14c/0x200
 [] SYSC_bind+0xe0/0x120
 [] ? __secure_computing+0x73/0x240
 [] ? __audit_syscall_exit+0x1e6/0x280
 [] ? __audit_syscall_entry+0xb4/0x110
 [] ? syscall_trace_enter+0x173/0x220
 [] SyS_bind+0xe/0x10
 [] tracesys+0xdd/0xe2
Code: 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 01 c3 55 83 e2 fe 0f 
b7 f2 48 89 e5 b8 00 80 00 00 eb 0d 66 0f 1f 44 00 00 f3 90 <83> e8 01 74
 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ossec-list] Re: syscheckd causing soft lockups

2017-03-07 Thread John Gelnaw 

to follow up to my own post-- First, the problem was indeed happening 
during ossec-rootcheck, but I was unable to determine what was failing.

Secondly, the affected servers all were at one time or another, exporting a 
CIFS or NFS share.  Disabling the share didn't prevent ossec-rootcheck from 
crashing.

Reading the docs on 2.9.0, I thought the "skip_nfs" option might be 
helpful, so I upgraded-- but the problem went away before I enabled 
skip_nfs.

So upgrading seems to have solved the problem, but I don't know why.

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ossec-list] Re: syscheckd causing soft lockups

2017-03-01 Thread Santiago Bassett 
That is probably rootcheck trying to detect system anomalies and kernel
level rootkits. It does it by comparing the output of netstat with its own
results binding ports to check if they are open.

Remember that syscheckd not only does FIM, but also Rootchecks (policy
monitoring checks and anomalies detection).

You can disable these checks using

no (under rootcheck section).

Other checks are  done to detect hiddent files or processes.

Complete documentation here:

http://ossec-docs.readthedocs.io/en/latest/manual/rootcheck/manual-rootcheck.html

You can also enable debug for syscheck in internal_options.conf file (so
you get to know better what it is doing)

I hope it helps,

Santiago.

On Wed, Mar 1, 2017 at 7:59 AM, John Gelnaw  wrote:

>
> Followup. ossec-syscheckd appears to be doing some bind operation:
>
>
> socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6
> bind(6, {sa_family=AF_INET, sin_port=htons(12310),
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> close(6) = 0
> socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6
> bind(6, {sa_family=AF_INET, sin_port=htons(12311),
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> close(6) = 0
> socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6
> bind(6, {sa_family=AF_INET, sin_port=htons(12312),
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> close(6) = 0
> socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6
> bind(6, {sa_family=AF_INET, sin_port=htons(12313),
> sin_addr=inet_addr("0.0.0.0")}, 16
> 2017 Mar 1 10:27:58 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck
> for 22s! [ossec-syscheckd:19286]
> 2017 Mar 1 10:28:26 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck
> for 22s! [ossec-syscheckd:19286]
> 2017 Mar 1 10:28:58 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck
> for 22s! [ossec-syscheckd:19286]
> 2017 Mar 1 10:29:26 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck
> for 22s! [ossec-syscheckd:19286]
> 2017 Mar 1 10:29:54 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck
> for 22s! [ossec-syscheckd:19286]
>
> My guess is that it's trying to bind, running out of available sockets,
> and waiting until a socket frees up (or forever, whichever comes first).
>
> Why would syscheckd be attempting to bind to 0.0.0.0, however?
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ossec-list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ossec-list+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ossec-list] Re: syscheckd causing soft lockups

2017-03-01 Thread John Gelnaw 

Followup. ossec-syscheckd appears to be doing some bind operation:


socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6 
bind(6, {sa_family=AF_INET, sin_port=htons(12310), 
sin_addr=inet_addr("0.0.0.0")}, 16) = 0 
close(6) = 0 
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6 
bind(6, {sa_family=AF_INET, sin_port=htons(12311), 
sin_addr=inet_addr("0.0.0.0")}, 16) = 0 
close(6) = 0 
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6 
bind(6, {sa_family=AF_INET, sin_port=htons(12312), 
sin_addr=inet_addr("0.0.0.0")}, 16) = 0 
close(6) = 0 
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6 
bind(6, {sa_family=AF_INET, sin_port=htons(12313), 
sin_addr=inet_addr("0.0.0.0")}, 16
2017 Mar 1 10:27:58 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck 
for 22s! [ossec-syscheckd:19286]
2017 Mar 1 10:28:26 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck 
for 22s! [ossec-syscheckd:19286] 
2017 Mar 1 10:28:58 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck 
for 22s! [ossec-syscheckd:19286] 
2017 Mar 1 10:29:26 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck 
for 22s! [ossec-syscheckd:19286] 
2017 Mar 1 10:29:54 ahc-www01 NMI watchdog: BUG: soft lockup - CPU#0 stuck 
for 22s! [ossec-syscheckd:19286]

My guess is that it's trying to bind, running out of available sockets, and 
waiting until a socket frees up (or forever, whichever comes first).

Why would syscheckd be attempting to bind to 0.0.0.0, however?

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.