Re: [ovirt-users] OSSEC reporting hidden processes
Well yes I am sure about the research I did :-) However, to your point, I didnt actually consider that and of course now clearly makes the most sense. Thanks! On Wed, Mar 22, 2017 at 9:57 AM, Yedidyah Bar Davidwrote: > On Wed, Mar 22, 2017 at 3:43 PM, Charles Kozler > wrote: > > Thanks for the feedback! > > > > From my research though it seems like it would take some effort to start > a > > process and not register it in /proc or at least, it would be by > intention > > to do so for that desired affect. > > Are you sure about that? > > What if the process is simply very short-lived? Wouldn't something like > OSSEC > wrongly count it as "hiding" simply because it died in the middle of being > investigated? E.g. vdsm runs many times instances of 'dd' to check that > some > file is readable (or writable), IIRC. > > > I guess my ask here would be why would > > ovirt do that? Is there a relative performance gain? What processes > inside > > ovirt would do such a thing? > > > > Appreciate the help > > > > On Wed, Mar 22, 2017 at 3:32 AM, Yedidyah Bar David > wrote: > >> > >> On Tue, Mar 21, 2017 at 7:54 PM, Charles Kozler > >> wrote: > >> > Unfortunately by the time I am able to SSH to the server and start > >> > looking > >> > around, that PID is no where to be found > >> > >> Even if you do this immediately when OSSEC finishes? > >> Do you get from it only a single pid? > >> > >> > > >> > So it seems something winds up in ovirt, runs, doesnt register in > /proc > >> > (I > >> > think even threads register themself in /proc), > >> > >> Now did some tests. Seems like they do, but are only "visible" if you > >> access them directly, not if you e.g. 'ls -l /proc'. > >> > >> > and then dies off > >> > > >> > Any ideas? > >> > >> No idea about your specific issue. Based on your above question, did > this: > >> > >> # for pid in $(seq 32768); do if kill -0 $pid 2>/dev/null && ! ls -1 > >> /proc | grep -qw $pid; then ps -e -T | grep -w $pid | awk '{print > >> $1}'; fi; done | sort -u | while read ppid; do echo number of threads: > >> $(ps -e -T | grep -w $ppid | wc -l) ps $ppid: $(ps -h -p $ppid); done > >> number of threads: 5 ps 1149: 1149 ? Ssl 0:23 /usr/bin/python -Es > >> /usr/sbin/tuned -l -P > >> number of threads: 3 ps 1151: 1151 ? Ssl 0:55 /usr/sbin/rsyslogd -n > >> number of threads: 2 ps 1155: 1155 ? Ssl 0:00 /usr/bin/ruby > >> /usr/bin/fluentd -c /etc/fluentd/fluent.conf > >> number of threads: 12 ps 1156: 1156 ? Ssl 4:49 /usr/sbin/collectd > >> number of threads: 16 ps 1205: 1205 ? Ssl 0:08 /usr/sbin/libvirtd > --listen > >> number of threads: 6 ps 1426: 1426 ? Sl 23:57 /usr/bin/ruby > >> /usr/bin/fluentd -c /etc/fluentd/fluent.conf > >> number of threads: 32 ps 3171: 3171 ? S >> /usr/share/vdsm/vdsmd > >> number of threads: 6 ps 3173: 3173 ? Ssl 8:48 python /usr/sbin/momd -c > >> /etc/vdsm/mom.conf > >> number of threads: 7 ps 575: 575 ? SLl 0:14 /sbin/multipathd > >> number of threads: 3 ps 667: 667 ? SLsl 0:09 /usr/sbin/dmeventd -f > >> number of threads: 2 ps 706: 706 ? S >> number of threads: 6 ps 730: 730 ? Ssl 0:00 /usr/lib/polkit-1/polkitd > >> --no-debug > >> number of threads: 3 ps 735: 735 ? Ssl 0:31 /usr/bin/python > >> /usr/bin/ovirt-imageio-daemon > >> number of threads: 4 ps 741: 741 ? S >> /usr/share/vdsm/supervdsmd --sockfile /var/run/vdsm/svdsm.sock > >> number of threads: 2 ps 743: 743 ? Ssl 0:00 /bin/dbus-daemon --system > >> --address=systemd: --nofork --nopidfile --systemd-activation > >> number of threads: 6 ps 759: 759 ? Ssl 0:00 /usr/sbin/gssproxy -D > >> number of threads: 5 ps 790: 790 ? SLsl 0:09 /usr/sbin/sanlock daemon > >> > >> (There are probably more efficient ways to do this, nvm). > >> > >> > > >> > On Tue, Mar 21, 2017 at 3:10 AM, Yedidyah Bar David > >> > wrote: > >> >> > >> >> On Mon, Mar 20, 2017 at 5:59 PM, Charles Kozler < > ckozler...@gmail.com> > >> >> wrote: > >> >> > Hi - > >> >> > > >> >> > I am wondering why OSSEC would be reporting hidden processes on my > >> >> > ovirt > >> >> > nodes? I run OSSEC across the infrastructure and multiple ovirt > >> >> > clusters > >> >> > have assorted nodes that will report a process is running but does > >> >> > not > >> >> > have > >> >> > an entry in /proc and thus "possible rootkit" alert is fired > >> >> > > >> >> > I am well aware that I do not have rootkits on these systems but am > >> >> > wondering what exactly inside ovirt is causing this to trigger? Or > >> >> > any > >> >> > ideas? Below is sample alert. All my google-fu turns up is that a > >> >> > process > >> >> > would have to **try** to hide itself from /proc, so curious what > this > >> >> > is > >> >> > inside ovirt. Thanks! > >> >> > > >> >> > - > >> >> > > >> >> > OSSEC HIDS Notification. > >> >> > 2017 Mar 20 11:54:47 > >> >> > > >> >> > Received From: (ovirtnode2.mydomain.com2) any->rootcheck > >> >> > Rule: 510 fired (level 7) ->
Re: [ovirt-users] OSSEC reporting hidden processes
On Wed, Mar 22, 2017 at 3:43 PM, Charles Kozlerwrote: > Thanks for the feedback! > > From my research though it seems like it would take some effort to start a > process and not register it in /proc or at least, it would be by intention > to do so for that desired affect. Are you sure about that? What if the process is simply very short-lived? Wouldn't something like OSSEC wrongly count it as "hiding" simply because it died in the middle of being investigated? E.g. vdsm runs many times instances of 'dd' to check that some file is readable (or writable), IIRC. > I guess my ask here would be why would > ovirt do that? Is there a relative performance gain? What processes inside > ovirt would do such a thing? > > Appreciate the help > > On Wed, Mar 22, 2017 at 3:32 AM, Yedidyah Bar David wrote: >> >> On Tue, Mar 21, 2017 at 7:54 PM, Charles Kozler >> wrote: >> > Unfortunately by the time I am able to SSH to the server and start >> > looking >> > around, that PID is no where to be found >> >> Even if you do this immediately when OSSEC finishes? >> Do you get from it only a single pid? >> >> > >> > So it seems something winds up in ovirt, runs, doesnt register in /proc >> > (I >> > think even threads register themself in /proc), >> >> Now did some tests. Seems like they do, but are only "visible" if you >> access them directly, not if you e.g. 'ls -l /proc'. >> >> > and then dies off >> > >> > Any ideas? >> >> No idea about your specific issue. Based on your above question, did this: >> >> # for pid in $(seq 32768); do if kill -0 $pid 2>/dev/null && ! ls -1 >> /proc | grep -qw $pid; then ps -e -T | grep -w $pid | awk '{print >> $1}'; fi; done | sort -u | while read ppid; do echo number of threads: >> $(ps -e -T | grep -w $ppid | wc -l) ps $ppid: $(ps -h -p $ppid); done >> number of threads: 5 ps 1149: 1149 ? Ssl 0:23 /usr/bin/python -Es >> /usr/sbin/tuned -l -P >> number of threads: 3 ps 1151: 1151 ? Ssl 0:55 /usr/sbin/rsyslogd -n >> number of threads: 2 ps 1155: 1155 ? Ssl 0:00 /usr/bin/ruby >> /usr/bin/fluentd -c /etc/fluentd/fluent.conf >> number of threads: 12 ps 1156: 1156 ? Ssl 4:49 /usr/sbin/collectd >> number of threads: 16 ps 1205: 1205 ? Ssl 0:08 /usr/sbin/libvirtd --listen >> number of threads: 6 ps 1426: 1426 ? Sl 23:57 /usr/bin/ruby >> /usr/bin/fluentd -c /etc/fluentd/fluent.conf >> number of threads: 32 ps 3171: 3171 ? S> /usr/share/vdsm/vdsmd >> number of threads: 6 ps 3173: 3173 ? Ssl 8:48 python /usr/sbin/momd -c >> /etc/vdsm/mom.conf >> number of threads: 7 ps 575: 575 ? SLl 0:14 /sbin/multipathd >> number of threads: 3 ps 667: 667 ? SLsl 0:09 /usr/sbin/dmeventd -f >> number of threads: 2 ps 706: 706 ? S> number of threads: 6 ps 730: 730 ? Ssl 0:00 /usr/lib/polkit-1/polkitd >> --no-debug >> number of threads: 3 ps 735: 735 ? Ssl 0:31 /usr/bin/python >> /usr/bin/ovirt-imageio-daemon >> number of threads: 4 ps 741: 741 ? S> /usr/share/vdsm/supervdsmd --sockfile /var/run/vdsm/svdsm.sock >> number of threads: 2 ps 743: 743 ? Ssl 0:00 /bin/dbus-daemon --system >> --address=systemd: --nofork --nopidfile --systemd-activation >> number of threads: 6 ps 759: 759 ? Ssl 0:00 /usr/sbin/gssproxy -D >> number of threads: 5 ps 790: 790 ? SLsl 0:09 /usr/sbin/sanlock daemon >> >> (There are probably more efficient ways to do this, nvm). >> >> > >> > On Tue, Mar 21, 2017 at 3:10 AM, Yedidyah Bar David >> > wrote: >> >> >> >> On Mon, Mar 20, 2017 at 5:59 PM, Charles Kozler >> >> wrote: >> >> > Hi - >> >> > >> >> > I am wondering why OSSEC would be reporting hidden processes on my >> >> > ovirt >> >> > nodes? I run OSSEC across the infrastructure and multiple ovirt >> >> > clusters >> >> > have assorted nodes that will report a process is running but does >> >> > not >> >> > have >> >> > an entry in /proc and thus "possible rootkit" alert is fired >> >> > >> >> > I am well aware that I do not have rootkits on these systems but am >> >> > wondering what exactly inside ovirt is causing this to trigger? Or >> >> > any >> >> > ideas? Below is sample alert. All my google-fu turns up is that a >> >> > process >> >> > would have to **try** to hide itself from /proc, so curious what this >> >> > is >> >> > inside ovirt. Thanks! >> >> > >> >> > - >> >> > >> >> > OSSEC HIDS Notification. >> >> > 2017 Mar 20 11:54:47 >> >> > >> >> > Received From: (ovirtnode2.mydomain.com2) any->rootcheck >> >> > Rule: 510 fired (level 7) -> "Host-based anomaly detection event >> >> > (rootcheck)." >> >> > Portion of the log(s): >> >> > >> >> > Process '24574' hidden from /proc. Possible kernel level rootkit. >> >> >> >> What do you get from: >> >> >> >> ps -eLf | grep -w 24574 >> >> >> >> Thanks, >> >> -- >> >> Didi >> > >> > >> >> >> >> -- >> Didi > > -- Didi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] OSSEC reporting hidden processes
Thanks for the feedback! >From my research though it seems like it would take some effort to start a process and not register it in /proc or at least, it would be by intention to do so for that desired affect. I guess my ask here would be why would ovirt do that? Is there a relative performance gain? What processes inside ovirt would do such a thing? Appreciate the help On Wed, Mar 22, 2017 at 3:32 AM, Yedidyah Bar Davidwrote: > On Tue, Mar 21, 2017 at 7:54 PM, Charles Kozler > wrote: > > Unfortunately by the time I am able to SSH to the server and start > looking > > around, that PID is no where to be found > > Even if you do this immediately when OSSEC finishes? > Do you get from it only a single pid? > > > > > So it seems something winds up in ovirt, runs, doesnt register in /proc > (I > > think even threads register themself in /proc), > > Now did some tests. Seems like they do, but are only "visible" if you > access them directly, not if you e.g. 'ls -l /proc'. > > > and then dies off > > > > Any ideas? > > No idea about your specific issue. Based on your above question, did this: > > # for pid in $(seq 32768); do if kill -0 $pid 2>/dev/null && ! ls -1 > /proc | grep -qw $pid; then ps -e -T | grep -w $pid | awk '{print > $1}'; fi; done | sort -u | while read ppid; do echo number of threads: > $(ps -e -T | grep -w $ppid | wc -l) ps $ppid: $(ps -h -p $ppid); done > number of threads: 5 ps 1149: 1149 ? Ssl 0:23 /usr/bin/python -Es > /usr/sbin/tuned -l -P > number of threads: 3 ps 1151: 1151 ? Ssl 0:55 /usr/sbin/rsyslogd -n > number of threads: 2 ps 1155: 1155 ? Ssl 0:00 /usr/bin/ruby > /usr/bin/fluentd -c /etc/fluentd/fluent.conf > number of threads: 12 ps 1156: 1156 ? Ssl 4:49 /usr/sbin/collectd > number of threads: 16 ps 1205: 1205 ? Ssl 0:08 /usr/sbin/libvirtd --listen > number of threads: 6 ps 1426: 1426 ? Sl 23:57 /usr/bin/ruby > /usr/bin/fluentd -c /etc/fluentd/fluent.conf > number of threads: 32 ps 3171: 3171 ? S /usr/share/vdsm/vdsmd > number of threads: 6 ps 3173: 3173 ? Ssl 8:48 python /usr/sbin/momd -c > /etc/vdsm/mom.conf > number of threads: 7 ps 575: 575 ? SLl 0:14 /sbin/multipathd > number of threads: 3 ps 667: 667 ? SLsl 0:09 /usr/sbin/dmeventd -f > number of threads: 2 ps 706: 706 ? S number of threads: 6 ps 730: 730 ? Ssl 0:00 /usr/lib/polkit-1/polkitd > --no-debug > number of threads: 3 ps 735: 735 ? Ssl 0:31 /usr/bin/python > /usr/bin/ovirt-imageio-daemon > number of threads: 4 ps 741: 741 ? S /usr/share/vdsm/supervdsmd --sockfile /var/run/vdsm/svdsm.sock > number of threads: 2 ps 743: 743 ? Ssl 0:00 /bin/dbus-daemon --system > --address=systemd: --nofork --nopidfile --systemd-activation > number of threads: 6 ps 759: 759 ? Ssl 0:00 /usr/sbin/gssproxy -D > number of threads: 5 ps 790: 790 ? SLsl 0:09 /usr/sbin/sanlock daemon > > (There are probably more efficient ways to do this, nvm). > > > > > On Tue, Mar 21, 2017 at 3:10 AM, Yedidyah Bar David > wrote: > >> > >> On Mon, Mar 20, 2017 at 5:59 PM, Charles Kozler > >> wrote: > >> > Hi - > >> > > >> > I am wondering why OSSEC would be reporting hidden processes on my > ovirt > >> > nodes? I run OSSEC across the infrastructure and multiple ovirt > clusters > >> > have assorted nodes that will report a process is running but does not > >> > have > >> > an entry in /proc and thus "possible rootkit" alert is fired > >> > > >> > I am well aware that I do not have rootkits on these systems but am > >> > wondering what exactly inside ovirt is causing this to trigger? Or any > >> > ideas? Below is sample alert. All my google-fu turns up is that a > >> > process > >> > would have to **try** to hide itself from /proc, so curious what this > is > >> > inside ovirt. Thanks! > >> > > >> > - > >> > > >> > OSSEC HIDS Notification. > >> > 2017 Mar 20 11:54:47 > >> > > >> > Received From: (ovirtnode2.mydomain.com2) any->rootcheck > >> > Rule: 510 fired (level 7) -> "Host-based anomaly detection event > >> > (rootcheck)." > >> > Portion of the log(s): > >> > > >> > Process '24574' hidden from /proc. Possible kernel level rootkit. > >> > >> What do you get from: > >> > >> ps -eLf | grep -w 24574 > >> > >> Thanks, > >> -- > >> Didi > > > > > > > > -- > Didi > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] OSSEC reporting hidden processes
On Tue, Mar 21, 2017 at 7:54 PM, Charles Kozlerwrote: > Unfortunately by the time I am able to SSH to the server and start looking > around, that PID is no where to be found Even if you do this immediately when OSSEC finishes? Do you get from it only a single pid? > > So it seems something winds up in ovirt, runs, doesnt register in /proc (I > think even threads register themself in /proc), Now did some tests. Seems like they do, but are only "visible" if you access them directly, not if you e.g. 'ls -l /proc'. > and then dies off > > Any ideas? No idea about your specific issue. Based on your above question, did this: # for pid in $(seq 32768); do if kill -0 $pid 2>/dev/null && ! ls -1 /proc | grep -qw $pid; then ps -e -T | grep -w $pid | awk '{print $1}'; fi; done | sort -u | while read ppid; do echo number of threads: $(ps -e -T | grep -w $ppid | wc -l) ps $ppid: $(ps -h -p $ppid); done number of threads: 5 ps 1149: 1149 ? Ssl 0:23 /usr/bin/python -Es /usr/sbin/tuned -l -P number of threads: 3 ps 1151: 1151 ? Ssl 0:55 /usr/sbin/rsyslogd -n number of threads: 2 ps 1155: 1155 ? Ssl 0:00 /usr/bin/ruby /usr/bin/fluentd -c /etc/fluentd/fluent.conf number of threads: 12 ps 1156: 1156 ? Ssl 4:49 /usr/sbin/collectd number of threads: 16 ps 1205: 1205 ? Ssl 0:08 /usr/sbin/libvirtd --listen number of threads: 6 ps 1426: 1426 ? Sl 23:57 /usr/bin/ruby /usr/bin/fluentd -c /etc/fluentd/fluent.conf number of threads: 32 ps 3171: 3171 ? S > On Tue, Mar 21, 2017 at 3:10 AM, Yedidyah Bar David wrote: >> >> On Mon, Mar 20, 2017 at 5:59 PM, Charles Kozler >> wrote: >> > Hi - >> > >> > I am wondering why OSSEC would be reporting hidden processes on my ovirt >> > nodes? I run OSSEC across the infrastructure and multiple ovirt clusters >> > have assorted nodes that will report a process is running but does not >> > have >> > an entry in /proc and thus "possible rootkit" alert is fired >> > >> > I am well aware that I do not have rootkits on these systems but am >> > wondering what exactly inside ovirt is causing this to trigger? Or any >> > ideas? Below is sample alert. All my google-fu turns up is that a >> > process >> > would have to **try** to hide itself from /proc, so curious what this is >> > inside ovirt. Thanks! >> > >> > - >> > >> > OSSEC HIDS Notification. >> > 2017 Mar 20 11:54:47 >> > >> > Received From: (ovirtnode2.mydomain.com2) any->rootcheck >> > Rule: 510 fired (level 7) -> "Host-based anomaly detection event >> > (rootcheck)." >> > Portion of the log(s): >> > >> > Process '24574' hidden from /proc. Possible kernel level rootkit. >> >> What do you get from: >> >> ps -eLf | grep -w 24574 >> >> Thanks, >> -- >> Didi > > -- Didi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] OSSEC reporting hidden processes
Unfortunately by the time I am able to SSH to the server and start looking around, that PID is no where to be found So it seems something winds up in ovirt, runs, doesnt register in /proc (I think even threads register themself in /proc), and then dies off Any ideas? On Tue, Mar 21, 2017 at 3:10 AM, Yedidyah Bar Davidwrote: > On Mon, Mar 20, 2017 at 5:59 PM, Charles Kozler > wrote: > > Hi - > > > > I am wondering why OSSEC would be reporting hidden processes on my ovirt > > nodes? I run OSSEC across the infrastructure and multiple ovirt clusters > > have assorted nodes that will report a process is running but does not > have > > an entry in /proc and thus "possible rootkit" alert is fired > > > > I am well aware that I do not have rootkits on these systems but am > > wondering what exactly inside ovirt is causing this to trigger? Or any > > ideas? Below is sample alert. All my google-fu turns up is that a process > > would have to **try** to hide itself from /proc, so curious what this is > > inside ovirt. Thanks! > > > > - > > > > OSSEC HIDS Notification. > > 2017 Mar 20 11:54:47 > > > > Received From: (ovirtnode2.mydomain.com2) any->rootcheck > > Rule: 510 fired (level 7) -> "Host-based anomaly detection event > > (rootcheck)." > > Portion of the log(s): > > > > Process '24574' hidden from /proc. Possible kernel level rootkit. > > What do you get from: > > ps -eLf | grep -w 24574 > > Thanks, > -- > Didi > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] OSSEC reporting hidden processes
On Mon, Mar 20, 2017 at 5:59 PM, Charles Kozlerwrote: > Hi - > > I am wondering why OSSEC would be reporting hidden processes on my ovirt > nodes? I run OSSEC across the infrastructure and multiple ovirt clusters > have assorted nodes that will report a process is running but does not have > an entry in /proc and thus "possible rootkit" alert is fired > > I am well aware that I do not have rootkits on these systems but am > wondering what exactly inside ovirt is causing this to trigger? Or any > ideas? Below is sample alert. All my google-fu turns up is that a process > would have to **try** to hide itself from /proc, so curious what this is > inside ovirt. Thanks! > > - > > OSSEC HIDS Notification. > 2017 Mar 20 11:54:47 > > Received From: (ovirtnode2.mydomain.com2) any->rootcheck > Rule: 510 fired (level 7) -> "Host-based anomaly detection event > (rootcheck)." > Portion of the log(s): > > Process '24574' hidden from /proc. Possible kernel level rootkit. What do you get from: ps -eLf | grep -w 24574 Thanks, -- Didi ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] OSSEC reporting hidden processes
Hi - I am wondering why OSSEC would be reporting hidden processes on my ovirt nodes? I run OSSEC across the infrastructure and multiple ovirt clusters have assorted nodes that will report a process is running but does not have an entry in /proc and thus "possible rootkit" alert is fired I am well aware that I do not have rootkits on these systems but am wondering what exactly inside ovirt is causing this to trigger? Or any ideas? Below is sample alert. All my google-fu turns up is that a process would have to **try** to hide itself from /proc, so curious what this is inside ovirt. Thanks! - OSSEC HIDS Notification. 2017 Mar 20 11:54:47 Received From: (ovirtnode2.mydomain.com2) any->rootcheck Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)." Portion of the log(s): Process '24574' hidden from /proc. Possible kernel level rootkit. --END OF NOTIFICATION ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users