Re: [Evergreen] 13.1: nscd consumes a lot of memory

2016-04-07 Thread Ruediger Meier 
On Thursday 07 April 2016, Christian Boltz wrote:
> Hello,
>
> Am Donnerstag, 7. April 2016, 09:57:17 CEST schrieb Ruediger Meier:
> > On Wednesday 06 April 2016, Christian Boltz wrote:
>
> ...
>
> > > You'll probably need to adjust the AppArmor profile a bit to
> > > allow that. Otherwise, nscd won't be able to restart itsself
> > > (which effectively means ignoring the paranoia mode).
> > >
> > > https://bugzilla.opensuse.org/show_bug.cgi?id=971790
> >
> > Thanks, good to know.
> >
> > There seems to be another bug
> > $ nscd --invalidate
> > does not to work.
> >
> > The only way to reset cache and stats is to set
> >   persistent   xyz   no
> > and rcnscd.restart.
>
> I Just tested this (on 13.1, and also on Tumbleweed), and it works.
>
> Does the syslog tell you anything about why --invalidate doesn't
> work?

No logs. Actually it looks like it works but does not clear anything:

$ nscd -g | grep -A20 "passwd cache:"
passwd cache:
yes  cache is enabled
 no  cache is persistent
 no  cache is shared
211  suggested size
 216064  total data pool size
184  used data pool size
600  seconds time to live for positive entries
 20  seconds time to live for negative entries
623  cache hits on positive entries
  0  cache hits on negative entries
 19  cache misses on positive entries
 24  cache misses on negative entries
 93% cache hit rate
  2  current number of cached values
 28  maximum number of cached values
  2  maximum chain length searched
  0  number of delays on rdlock
  0  number of delays on wrlock
  0  memory allocations failed
$ nscd -i passwd
$ nscd -g | grep -A20 "passwd cache:"
passwd cache:
yes  cache is enabled
 no  cache is persistent
 no  cache is shared
211  suggested size
 216064  total data pool size
  0  used data pool size
600  seconds time to live for positive entries
 20  seconds time to live for negative entries
623  cache hits on positive entries
  0  cache hits on negative entries
 19  cache misses on positive entries
 24  cache misses on negative entries
 93% cache hit rate
  0  current number of cached values
 28  maximum number of cached values
  2  maximum chain length searched
  0  number of delays on rdlock
  0  number of delays on wrlock
  0  memory allocations failed


Only restart clears the cache (and only if you have set "persistent 
passwd no"):

$ rcnscd restart
$ nscd -g | grep -A20 "passwd cache:"
passwd cache:
yes  cache is enabled
 no  cache is persistent
 no  cache is shared
211  suggested size
 216064  total data pool size
  0  used data pool size
600  seconds time to live for positive entries
 20  seconds time to live for negative entries
  0  cache hits on positive entries
  0  cache hits on negative entries
  0  cache misses on positive entries
  0  cache misses on negative entries
  0% cache hit rate
  0  current number of cached values
  0  maximum number of cached values
  0  maximum chain length searched
  0  number of delays on rdlock
  0  number of delays on wrlock
  0  memory allocations failed


> Does /var/log/audit/audit.log contain any denials? (I doubt 
> that AppArmor restrictions are involved here, but just to be sure.)
>
> > How would I disable appamor system wide?
>
> Like every other service - rcapparmor stop

Ooops ...

> [1] 

Ok, I knew that there was something special ;)

> However I hope you won't do this. The better way is usually to

Yep, just for quick, painless testing.

> - switch the profile to complain/learning mode:   aa-complain nscd
> - let the service run for a while
> - run   aa-logprof   to update the profile
> - enforce the profile again:   aa-enforce nscd
>
> BTW: You might be interested in my AppArmor Crash Course:
> http://blog.cboltz.de/archives/65-openSUSE-conference.html
> (PDFs linked at the end of the article)
>
> I'll also give an updated version of this talk at the openSUSE
> conference (assuming my proposal gets accepted).
>
___
Evergreen mailing list
Evergreen@lists.rosenauer.org
http://lists.rosenauer.org/mailman/listinfo/evergreen

Re: [Evergreen] 13.1: nscd consumes a lot of memory

2016-04-06 Thread Ruediger Meier 
On Wednesday 06 April 2016, Patrick Shanahan wrote:
> * Ruediger Meier <sweet_...@gmx.de> [04-06-16 10:45]:
> > Hi,
> >
> > Since a few weeks I see nscd running crazy on 13.1, like this
> >
> > USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+
> > COMMAND nscd  20   0 5265008 1.626g652 S 0.000 5.185  
> > 0:30.32 nscd
> >
> > I guess 1.6G would be enough to cache all existing zones worldwide.
> > What could be the problem?
>
> odd, mine has been up 261 days and:

Could you show me your stats? I wonder if you have some reasonable cache 
hit rates.
$ nscd -g


> USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME
> COMMAND nscd  2785  0.0  0.1 1001076 11276 ?   Ssl  Feb24 
> 16:59 /usr/sbin/nscd --foreground
>
> but I am still on kernel-desktop-3.11.10-29.1.x86_64
> nscd-2.18-4.44.1.x86_64

I assume that it has something to do with recent updates (kernel, 
systemd?). Maybe my NIS setup is also a bit unusual but this is the 
reason why I run nscd at all.

Anyways it's for sure a bug. It should never use much more than
(5 * 32M) = 160M:

$ grep max-db /etc/nscd.conf
#   max-db-size  
max-db-size passwd  33554432
max-db-size group   33554432
max-db-size hosts   33554432
max-db-size services33554432
max-db-size netgroup33554432


I really hate that nscd. I've had also problems with it a few years ago 
on 11.4. (crashs, 100% CPU, ...). I wonder why the glibc people do not 
fix it. Instead they have added a "paranoia" mode to restart it 
regularly:
http://serverfault.com/questions/463648/why-would-nscd-use-a-huge-amount-of-ram

I'll try /etc/nscd.conf:
paranoiayes
restart-interval3600

cu,
Rudi
___
Evergreen mailing list
Evergreen@lists.rosenauer.org
http://lists.rosenauer.org/mailman/listinfo/evergreen

Re: [Evergreen] 13.1: nscd consumes a lot of memory

2016-04-06 Thread Ruediger Meier 
On Wednesday 06 April 2016, Patrick Shanahan wrote:
> * Ruediger Meier <sweet_...@gmx.de> [04-06-16 16:03]:
> > On Wednesday 06 April 2016, Patrick Shanahan wrote:
> > > * Ruediger Meier <sweet_...@gmx.de> [04-06-16 13:07]:
> > > > On Wednesday 06 April 2016, Patrick Shanahan wrote:
> > > > > * Ruediger Meier <sweet_...@gmx.de> [04-06-16 10:45]:
> > > > > > Hi,
> > > > > >
> > > > > > Since a few weeks I see nscd running crazy on 13.1, like
> > > > > > this
> > > > > >
> > > > > > USER  PR  NIVIRTRESSHR S  %CPU  %MEM
> > > > > > TIME+ COMMAND nscd  20   0 5265008 1.626g652 S
> > > > > > 0.000 5.185 0:30.32 nscd
> > > > > >
> > > > > > I guess 1.6G would be enough to cache all existing zones
> > > > > > worldwide. What could be the problem?
> > > > >
> > > > > odd, mine has been up 261 days and:
> > > >
> > > > Could you show me your stats? I wonder if you have some
> > > > reasonable cache hit rates.
> > > > $ nscd -g
> > >
> > > nscd configuration:
> > >
> > >   0  server debug level
> > >  42d  3h 27m 41s  server runtime
> >
> > So it was restarted during uptime. Because of libc update, or
> > crash?
>
> must have been libc update, I don't recall now,


> but not crash or 
> uptime would be less/reset ...

Ah, I've ment nscd crash/restart, not whole system crash. Could be that 
OOM killer killed it. This happened on my other machine where nscd 
consumed more than 8GB! memory.

cu,
Rudi 
___
Evergreen mailing list
Evergreen@lists.rosenauer.org
http://lists.rosenauer.org/mailman/listinfo/evergreen

Re: [Evergreen] 13.1: nscd consumes a lot of memory

2016-04-07 Thread Ruediger Meier 
On Wednesday 06 April 2016, Christian Boltz wrote:
> Hello,
>
> Am Mittwoch, 6. April 2016, 18:07:26 CEST schrieb Ruediger Meier:
> ...
>
> > I really hate that nscd. I've had also problems with it a few years
> > ago on 11.4. (crashs, 100% CPU, ...). I wonder why the glibc people
> > do not fix it. Instead they have added a "paranoia" mode to restart
> > it regularly:
> > http://serverfault.com/questions/463648/why-would-nscd-use-a-huge-a
> >mou nt-of-ram
> >
> > I'll try /etc/nscd.conf:
> > paranoiayes
> > restart-interval3600
>
> You'll probably need to adjust the AppArmor profile a bit to allow
> that. Otherwise, nscd won't be able to restart itsself (which
> effectively means ignoring the paranoia mode).
>
> https://bugzilla.opensuse.org/show_bug.cgi?id=971790

Thanks, good to know.

There seems to be another bug
$ nscd --invalidate
does not to work.

The only way to reset cache and stats is to set 
  persistent   xyz   no
and rcnscd.restart.


> The profile needs to allow reading /proc/$pid/cmdline, which means
> you'll need to add the rule
>   @{PROC}/@{pid}/cmdline r,
>
> The easiest way is probably:
> echo '@{PROC}/@{pid}/cmdline r,' >>
> /etc/apparmor.d/local/usr.sbin.nscd rcapparmor reload

How would I disable appamor system wide?

> This is fixed in the upstream profile (including the upcoming 2.9.3
> and 2.10.1 releases), but I slightly doubt this is worth an update
> for Evergreen ;-)
>
>
> Regards,
>
> Christian Boltz
___
Evergreen mailing list
Evergreen@lists.rosenauer.org
http://lists.rosenauer.org/mailman/listinfo/evergreen

Re: [Evergreen] Where's the last 3.12 kernel update for 13.1

2016-04-26 Thread Ruediger Meier 
On Tuesday 26 April 2016, Marcus Meissner wrote:
> On Tue, Apr 26, 2016 at 03:12:57PM +0200, Michal Kubecek wrote:
> > On úterý 26. dubna 2016 15:02 Marcus Meissner wrote:
> > > We have a fresh 3.12 kernel in QA, at
> > > http://download.opensuse.org/repositories/openSUSE:/Maintenance:/
> > >4989/openSUSE_Leap_42.1_Update/
> > >
> > > Could people test it?
> >
> > Should be
> >
> > http://download.opensuse.org/repositories/openSUSE:/Maintenance:/49
> >66/openSUSE_13.1_Update/

BTW one thing I was always wondering about. What is the most simple
way to install such non-offcial patch without adding that Maintenance
repo. Do we have something like

zypper patch -r 
http://download.opensuse.org/repositories/openSUSE:/Maintenance:/4966/openSUSE_13.1_Update/


cu,
Rudi
___
Evergreen mailing list
Evergreen@lists.rosenauer.org
http://lists.rosenauer.org/mailman/listinfo/evergreen

[Evergreen] FYI nvidia update for 13.1 kernels

2016-04-26 Thread Ruediger Meier 
Hi,

I'm using a script to update nvidia modules for the new 3.12 kernels.
Maybe also useful for other's, here it is:

---
#!/bin/bash

# This script compiles and installs the nvidia kernel modules for
# non-official SUSE kernels to let it work with SUSE's nvidia
# packages without downloading drivers manually.
#
# Requirements:
#   - kernel-desktop-devel rpm (matching the new target kernel)
#   - nvidia kmp rpms (matchine the old official SUSE kernel)
#
# You must set the target kernel version here, we don't want to
# guess and use the running or last installed kernel!
arch=x86_64
kver=3.12.57-44
flavor=desktop

function nvidia_ko() {
local ver=$1
local kver=$2
make -C /usr/src/linux-$kver-obj/$arch/$flavor \
modules \
M=/usr/src/kernel-modules/nvidia-$ver-$flavor \
SYSSRC=/usr/src/linux-$kver \
SYSOUT=/usr/src/linux-$kver-obj/$arch/$flavor || return

rm -rf /tmp/nvidia_ko
cp -a /usr/src/kernel-modules/nvidia-$ver-$flavor /tmp/nvidia_ko || return
pushd /tmp/nvidia_ko

make -f Makefile \
nv-linux.o \
SYSSRC=/usr/src/linux-$kver \
SYSOUT=/usr/src/linux-$kver-obj/$arch/$flavor || return
popd

#kver=3.11.6-4

install -m 755 -d /lib/modules/$kver-$flavor/updates
install -m 644 /usr/src/kernel-modules/nvidia-$ver-$flavor/nvidia.ko \
/lib/modules/$kver-$flavor/updates
depmod $kver-$flavor

# TODO weak update support
#nvr=nvidia-gfxG03-kmp-desktop-340.96_k3.11.6_4-40.1
#wm2=/usr/lib/module-init-tools/weak-modules2
#if [ -x $wm2 ]; then
#   /bin/bash -${-/e/} $wm2 --add-kmp $nvr
#fi
}

function nvidia_uvm_ko() {
local ver=$1
local kver=$2

rm -rf /tmp/nvidia-uvm_ko
cp -a /usr/src/kernel-modules/nvidia-uvm-$ver-$flavor /tmp/nvidia-uvm_ko || 
return
pushd /tmp/nvidia-uvm_ko/rm

make module \
SYSSRC=/usr/src/linux-$kver 
SYSOUT=/usr/src/linux-$kver-obj/$arch/$flavor \
KBUILD_EXTMOD=/tmp/nvidia-uvm_ko/rm
popd

cp -f /tmp/nvidia-uvm_ko/rm/Module.symvers /tmp/nvidia-uvm_ko/Module.symvers

make -C /usr/src/linux-$kver-obj/$arch/$flavor \
modules \
M=/tmp/nvidia-uvm_ko \
SYSSRC=/usr/src/linux-$kver \
SYSOUT=/usr/src/linux-$kver-obj/$arch/$flavor \
RM_OUT_DIR=/tmp/nvidia-uvm_ko/rm

#kver=3.11.6-4

install -m 755 -d /lib/modules/$kver-$flavor/updates
install -m 644 /tmp/nvidia-uvm_ko/nvidia-uvm.ko \
/lib/modules/$kver-$flavor/updates
depmod $kver-$flavor

#TODO weak update support
#nvr=nvidia-uvm-gfxG03-kmp-desktop-340.96_k3.11.6_4-40.1
#wm2=/usr/lib/module-init-tools/weak-modules2
#if [ -x $wm2 ]; then
#   /bin/bash -${-/e/} $wm2 --add-kmp $nvr
#fi
}

read name ver_nvidia rest <<<$(rpm -qa --qf "%{NAME} %{VERSION}\n" | grep 
"nvidia-gfx.*kmp")
ver_nvidia=${ver_nvidia%%_*}
read name ver_nvidia_uvm rest <<<$(rpm -qa --qf "%{NAME} %{VERSION}\n" | grep 
"nvidia-uvm-gfx.*kmp")
ver_nvidia_uvm=${ver_nvidia_uvm%%_*}

if [ -n "$ver_nvidia" ]; then
echo "## rebuild nvidia.ko $ver_nvidia for kernel $kver"
nvidia_ko $ver_nvidia $kver
else
echo "## no nvidia.ko installed - nothing to do"
fi

if [ -n "$ver_nvidia_uvm" ]; then
echo "## rebuild nvidia_uvm.ko $ver_nvidia for kernel $kver"
nvidia_uvm_ko $ver_nvidia_uvm $kver
else
echo "## no nvidia_uvm.ko installed - nothing to do"
fi
-
___
Evergreen mailing list
Evergreen@lists.rosenauer.org
http://lists.rosenauer.org/mailman/listinfo/evergreen

Re: [Evergreen] Continuing evergreen with Leap42.2

2017-01-31 Thread Ruediger Meier 
On Monday 30 January 2017, Greg Freemyer wrote:
> On Mon, Jan 30, 2017 at 11:28 AM, reinhard hennig
>
>  wrote:
> > my administration providing 12.000 Linux clients would appreciate
> > continueing evergreen with Leap42.2, because it is planned
> > to switch to leap 42.2 (13.2 at the moment) in a couple of months.
> >
> > That would be great.
>
> I don't know if it is working out, but the fundamental concept of
> Leap is that the 42.x series is easy to upgrade between.  In theory,
> 42.2 is 42.1 with a major service pack.
>
> Major changes will be held back until 43.x.

That's the theory. In practice seems that any packager/maintainer 
changes their packages however they want.

For example one of my packages (sbcl) was even not building anymore for 
42,2 although it was fine for 42.1. I've asked to revert the other 
incompatible change which broke my package but no chance ... I've had 
to change my package too otherwise they would have even removed my 
package from 42.2! Even this was a known issue, avoiding or reverting 
incompatible changes is obviously not important for Leap.

42.2 got a lot upgrades for no reason. Ok, the desktop stuff is always 
f**ked up anyways and I gave up to complain about that, But even things 
like texlive (about 6000 packages) got an update from version 2013 to 
2015. Why? Because just one user asked about that on mailing list ...

> So the need for Evergreen is much reduced with Leap.  Maybe you can
> ask on the main lists if that desire is working out for people.  I
> never had a Leap 42.1 machine and I only have one 42.2 machine, so I
> can't speak to how the concept is working.
>
> I can say I heard very few complaints from people seeing regressions
> when moving from 42.1 to 42.2

Yes it worked quiet good and I have to admit that this time I even liked 
the kernel upgrade, etc. because our hardware (Skylake desktop and 
laptops with intel graphics) was "too new" for 42.1. But I've had 
expected it and had 42,1 only running on a few test machines. The other 
13.1 machines are re-installed directly with 42.2. My users will not 
have to deal with annoying upgrades once per year ...

In general I don't see a major advantage for leap vs the old release 
model. I would be happy to stick with 42.2 for the next years (unless 
somebody stops these random package upgrades in the leap series.)

cu,
Rudi
___
Evergreen mailing list
Evergreen@lists.rosenauer.org
http://lists.rosenauer.org/mailman/listinfo/evergreen

Re: [Evergreen] Continuing evergreen with Leap42.2

2017-02-23 Thread Ruediger Meier 
On Thursday 23 February 2017, Carlos E. R. wrote:
> El 2017-02-22 a las 11:37 +0100, Ruediger Meier escribió:
> >> For example one of my packages (sbcl) was even not building
> >> anymore for 42,2 although it was fine for 42.1. I've asked to
> >> revert the other incompatible change which broke my package but no
> >> chance ... I've had to change my package too otherwise they would
> >> have even removed my package from 42.2! Even this was a known
> >> issue, avoiding or reverting incompatible changes is obviously not
> >> important for Leap.
> >
> > ... and it happened again. They upgraded 42.3 to texlive-2016 and
> > it breaks again. Evergreen is still needed.
>
> That will be solved eventually, I hope.
>
> What worries me is that we got 2 kernel updates in a month. Several
> updates this month that require a reboot (systemd, apparmor...). Not
> good for server uptime.


You DO NOT NEED to reboot after kernel update. Nowadays the old kernel 
should be still installed in parallel so module loading still works 
without reboot.

You SHOULD reboot as soon as possible if the kernel update fixes 
security bugs _and_ if you have local users which you can't trust. For 
example if you run 100 workstations with 1000 users.

cu,
Rudi
___
Evergreen mailing list
Evergreen@lists.rosenauer.org
http://lists.rosenauer.org/mailman/listinfo/evergreen

Re: [Evergreen] Continuing evergreen with Leap42.2

2017-02-23 Thread Ruediger Meier 
On Thursday 23 February 2017, Carlos E. R. wrote:
> On 2017-02-23 13:46, Ruediger Meier wrote:
> > On Thursday 23 February 2017, Carlos E. R. wrote:
> >> El 2017-02-22 a las 11:37 +0100, Ruediger Meier escribió:
> >>> ... and it happened again. They upgraded 42.3 to texlive-2016 and
> >>> it breaks again. Evergreen is still needed.
> >>
> >> That will be solved eventually, I hope.
> >>
> >> What worries me is that we got 2 kernel updates in a month.
> >> Several updates this month that require a reboot (systemd,
> >> apparmor...). Not good for server uptime.
> >
> > You DO NOT NEED to reboot after kernel update. Nowadays the old
> > kernel should be still installed in parallel so module loading
> > still works without reboot.
>
> If I do that, then I would simply not apply the update at all. Far
> easier and more stable.


Hm, I never care about updates. I get them automatically everyday and
reboot whenever I want. I think that's more simple. Regarding stability
I don't think it makes any difference whether you install a kernel
update 5 minutes or 5 months before reboot.

BTW very often even the "non-reboot" updates require manually
restarting affected programs. Otherwise you would still use the
old versions too.

For example, below you see all the programs which are still using
vulnerable openssl. The fact that it's still running the old kernel
is my smallest problem ;)

glaukos:~ # uptime
 14:10pm  up 14 days 20:26,  1 user,  load average: 3.34, 3.32, 3.32

glaukos:~ # rpm -qa --last | head
mariadb-10.0.29-18.1.x86_64   Tue 21 Feb 2017 01:00:13 AM CET
mariadb-errormessages-10.0.29-18.1.x86_64 Tue 21 Feb 2017 01:00:11 AM CET
mariadb-client-10.0.29-18.1.x86_64Tue 21 Feb 2017 01:00:11 AM CET
openssl-1.0.2j-4.1.x86_64 Tue 21 Feb 2017 01:00:10 AM CET
libmysqlclient18-10.0.29-18.1.x86_64  Tue 21 Feb 2017 01:00:10 AM CET
expat-2.1.0-19.1.x86_64   Tue 21 Feb 2017 01:00:10 AM CET
libseccomp2-2.3.1-3.1.x86_64  Tue 21 Feb 2017 01:00:09 AM CET
libopenssl1_0_0-1.0.2j-4.1.x86_64 Tue 21 Feb 2017 01:00:09 AM CET
libexpat1-2.1.0-19.1.x86_64   Tue 21 Feb 2017 01:00:09 AM CET
kernel-default-4.4.46-11.1.x86_64 Wed 15 Feb 2017 01:00:13 AM CET

glaukos:~ # uname -r
4.4.36-8-default

glaukos:~ # zypper ps
The following running processes use deleted files:

PID   | PPID | UID  | User   | Command | Service | Files
--+--+--++-+-+
1 | 0| 0| root   | systemd | | 
/usr/lib64/libseccomp.so.2.3.0
  |  |  || | | 
/lib64/libapparmor.so.1.3.0
1077  | 0| 0| root   | systemd | | 
/usr/lib64/libseccomp.so.2.3.0
  |  |  || | | 
/lib64/libapparmor.so.1.3.0
1152  | 1| 499  | messagebus | dbus-daemon | dbus| 
/usr/lib64/libexpat.so.1.6.0
1635  | 1| 492  | nagios | nrpe| nrpe| 
/lib64/libssl.so.1.0.0
  |  |  || | | 
/lib64/libcrypto.so.1.0.0
1636  | 1| 0| root   | sshd| sshd| 
/lib64/libcrypto.so.1.0.0
1721  | 1| 44   | named  | named   | named   | 
/lib64/libcrypto.so.1.0.0
  |  |  || | | 
/usr/lib64/libmysqlclient.so.18.0.0
  |  |  || | | 
/usr/lib64/libxml2.so.2.9.4
  |  |  || | | 
/lib64/libssl.so.1.0.0
1743  | 1| 0| root   | nmbd| nmb | 
/lib64/libssl.so.1.0.0
  |  |  || | | 
/lib64/libcrypto.so.1.0.0
1807  | 1| 0| root   | smbd| smb | 
/lib64/libssl.so.1.0.0
  |  |  || | | 
/lib64/libcrypto.so.1.0.0
1808  | 1807 | 0| root   | smbd| smb | 
/lib64/libssl.so.1.0.0
  |  |  || | | 
/lib64/libcrypto.so.1.0.0
1809  | 1807 | 0| root   | smbd| smb | 
/lib64/libssl.so.1.0.0
  |  |  || | | 
/lib64/libcrypto.so.1.0.0
1855  | 1807 | 0| root   | smbd| smb | 
/lib64/libssl.so.1.0.0
  |  |  || | | 
/lib64/libcrypto.so.1.0.0
2889  | 1| 1000 | rudi   | systemd | | 
/usr/lib64/libseccomp.so.2.3.0
  |  |  || | | 
/lib64/libapparmor.so.1.3.0
2893  | 0| 1000 | rudi   | systemd | | 
/usr/lib64/libseccomp.so.2.3.0
  |  |  || | | 
/lib64/libapparmor.so.1.3.0
14248 | 1807 | 0| root