Re: nscd not caching
Bezüglich Eric van Gyzen's Nachricht vom 19.08.2014 15:39 (localtime): > On 08/19/2014 09:14, Harald Schmalzbauer wrote: >> … >>> At least that's what we found in the freebsd.org cluster. nss-pam-ldapd >>> was >>> two or three orders of magnitude more usable and got rid of nscd in the >>> process. >>> >>> For us, nscd "worked", but it just didn't save much effort because it was a >>> per-uid cache. ie: if "jkh" had just caused a ldap search, and "peter" >>> repeated it, it had to be done again from scratch. >>> >>> The downside for nss-pam-ldapd was that it uses a non-extensible wire >>> protocol >>> and didn't have room for bsd-style login classes. >> This exactly refelcts my experiences too, which is why I'm wondering if >> net/nss-pam-ldapd is a serious base candidate. >> When nscd showed up (arround 7.0-Release if I remember correctly), it >> was a big and highly appreciated improovement for me, reducing >> interactivity lags of gnome e.g. by at least a factor of 4 for usual >> desktop user tasks when user database was LDAP driven. >> At that time there were rumors that FreeBSD needs LDAP user-database >> support, but with the glitches of net/nss_ldap, it seemed that there's >> no ready-to-implement solution at that time. >> Things changed completely with net/nss-pam-ldapd. Haven't had any >> negative experiences with single-LDAP backend networks. Haven't had big >> environments yet either, but I think it's time to think about >> base-LDAP-support again. net/nss-pam-ldapd is GPL licensed, so I guess >> it won't get into base, but it was a great template, is it? > +1 for nss-pam-ldapd. We were using nss_ldap+pam_ldap, but switched to > nss-pam-ldapd. It's much faster and very reliable. We have several > multi-user FreeBSD systems (build servers, doing lots of lookups), > dozens of concurrent users and hundreds of total users, and Active > Directory servers. > > The way nss_ldap links the LDAP libraries into every process is not just > inefficient: it can be fatal. Thunderbird includes (or formerly > included?) its own private LDAP libraries. These conflicted with those > used by nss_ldap, so that Thunderbird would often crash. I don't know > if this is still a problem, but it's not /my/ problem anymore. > > As for the base system, "pkg install nss-pam-ldapd" is embarrassingly > easy and /much/ easier than adding to the base system. 'pkg install' is incredibly convenient these days, for sure, but in my world, hosts that don't need internet acces don't have internet access (4 out of 5). To make things worse, nfs exports aren't root-mapped (other than the default nobody:nobody), so I quiet often have the case that I have to provide net/nss-pam-ldapd via ISO-9660 image (for VMguests) or CD-media. That's not so convenient. Another limitation of 'pkg install' is that I can't influence what to install. Sometimes I want nss_ldap without pam_ldap. Therefore I'd need a compiler (somthing my production machines don't have) and ports, which in my case can't be fetched from internet nor from the NFS server (the latter has to be entered as LDAP user…) That's why I'd love to see base system LDAP support – I think it's very important to be able to setup a network computer in networks, which aren't interconnected with other networks/internet; these days more important ever since possible… -Harry signature.asc Description: OpenPGP digital signature
Re: nscd not caching
On 08/19/2014 09:14, Harald Schmalzbauer wrote: > Bezüglich Peter Wemm's Nachricht vom 17.08.2014 19:18 (localtime): >> On Sunday 17 August 2014 15:22:02 O. Hartmann wrote: >>> Am Sun, 17 Aug 2014 13:09:10 + >>> >>> "Eggert, Lars" schrieb: Nobody using nscd? Really? >>> I can only speak for myself and I stopped using nscd since the support is >>> crap. >>> >>> A while ago (t > 1 1/2 years) I realised within a OpenLDAP environment, that >>> when nscd is running, sometimes the system simple "forgets" about root - >>> this is painful while installing/updating ports and getting interrupted >>> with a weird error "unknown user root". >>> >>> nscd is supposed to be used in large environments where the cost for a user >>> lookup, like in OpenLDAP, is worse. But obviously FreeBSD isn't used in >>> that large environments with OpenLDAP and I'm wondering what the purpose of >>> nscd is. >> The other problem is that net/nss_ldap and security/pam_ldap have kind of >> been >> left behind for performance and robustness. People who use this seriousy >> tend >> to discover net/nss-pam-ldapd fairly quickly which has its own caching proxy >> and eliminates the need for nscd. >> >> With nss_ldap and pam_ldap, the openldap client libraries and startup cost >> is >> linked into every single binary that uses it. >> >> WIth pam-nss-ldapd, it's a simple unix domain socket (with peercred >> authentication) and no heavy-weight libraries in the consumers. The proxy on >> the other end of the socket keeps a ldap connection open (with an idle >> timeout). The whole thing was vastly more robust and efficient. >> >> At least that's what we found in the freebsd.org cluster. nss-pam-ldapd was >> two or three orders of magnitude more usable and got rid of nscd in the >> process. >> >> For us, nscd "worked", but it just didn't save much effort because it was a >> per-uid cache. ie: if "jkh" had just caused a ldap search, and "peter" >> repeated it, it had to be done again from scratch. >> >> The downside for nss-pam-ldapd was that it uses a non-extensible wire >> protocol >> and didn't have room for bsd-style login classes. > This exactly refelcts my experiences too, which is why I'm wondering if > net/nss-pam-ldapd is a serious base candidate. > When nscd showed up (arround 7.0-Release if I remember correctly), it > was a big and highly appreciated improovement for me, reducing > interactivity lags of gnome e.g. by at least a factor of 4 for usual > desktop user tasks when user database was LDAP driven. > At that time there were rumors that FreeBSD needs LDAP user-database > support, but with the glitches of net/nss_ldap, it seemed that there's > no ready-to-implement solution at that time. > Things changed completely with net/nss-pam-ldapd. Haven't had any > negative experiences with single-LDAP backend networks. Haven't had big > environments yet either, but I think it's time to think about > base-LDAP-support again. net/nss-pam-ldapd is GPL licensed, so I guess > it won't get into base, but it was a great template, is it? +1 for nss-pam-ldapd. We were using nss_ldap+pam_ldap, but switched to nss-pam-ldapd. It's much faster and very reliable. We have several multi-user FreeBSD systems (build servers, doing lots of lookups), dozens of concurrent users and hundreds of total users, and Active Directory servers. The way nss_ldap links the LDAP libraries into every process is not just inefficient: it can be fatal. Thunderbird includes (or formerly included?) its own private LDAP libraries. These conflicted with those used by nss_ldap, so that Thunderbird would often crash. I don't know if this is still a problem, but it's not /my/ problem anymore. As for the base system, "pkg install nss-pam-ldapd" is embarrassingly easy and /much/ easier than adding to the base system. Eric ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: nscd not caching
On 2014-8-19, at 13:54, Daniel Braniss wrote: > > I know that this a bit late but have you ever considered Hesiod? it uses > DNS/txt. > we have been using it since the days when BSDi had no NIS support and haven’t > seen a ypserver not responding since :-) I don't control the master NIS infrastructure, I just want to use it (with a server that is unfortunately 25ms away). We will move to LDAP at some time, but in the meantime, a functioning nscd would be nice. Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: nscd not caching
On 2014-8-18, at 20:23, John-Mark Gurney wrote: > Why not run a local slave on your server? I am trying to get one set up. It requires a change request to our organization's IT, which is, ahem, not always lightning fast. Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: nscd not caching
Bezüglich Peter Wemm's Nachricht vom 17.08.2014 19:18 (localtime): > On Sunday 17 August 2014 15:22:02 O. Hartmann wrote: >> Am Sun, 17 Aug 2014 13:09:10 + >> >> "Eggert, Lars" schrieb: >>> Nobody using nscd? Really? >> I can only speak for myself and I stopped using nscd since the support is >> crap. >> >> A while ago (t > 1 1/2 years) I realised within a OpenLDAP environment, that >> when nscd is running, sometimes the system simple "forgets" about root - >> this is painful while installing/updating ports and getting interrupted >> with a weird error "unknown user root". >> >> nscd is supposed to be used in large environments where the cost for a user >> lookup, like in OpenLDAP, is worse. But obviously FreeBSD isn't used in >> that large environments with OpenLDAP and I'm wondering what the purpose of >> nscd is. > The other problem is that net/nss_ldap and security/pam_ldap have kind of > been > left behind for performance and robustness. People who use this seriousy > tend > to discover net/nss-pam-ldapd fairly quickly which has its own caching proxy > and eliminates the need for nscd. > > With nss_ldap and pam_ldap, the openldap client libraries and startup cost is > linked into every single binary that uses it. > > WIth pam-nss-ldapd, it's a simple unix domain socket (with peercred > authentication) and no heavy-weight libraries in the consumers. The proxy on > the other end of the socket keeps a ldap connection open (with an idle > timeout). The whole thing was vastly more robust and efficient. > > At least that's what we found in the freebsd.org cluster. nss-pam-ldapd was > two or three orders of magnitude more usable and got rid of nscd in the > process. > > For us, nscd "worked", but it just didn't save much effort because it was a > per-uid cache. ie: if "jkh" had just caused a ldap search, and "peter" > repeated it, it had to be done again from scratch. > > The downside for nss-pam-ldapd was that it uses a non-extensible wire > protocol > and didn't have room for bsd-style login classes. This exactly refelcts my experiences too, which is why I'm wondering if net/nss-pam-ldapd is a serious base candidate. When nscd showed up (arround 7.0-Release if I remember correctly), it was a big and highly appreciated improovement for me, reducing interactivity lags of gnome e.g. by at least a factor of 4 for usual desktop user tasks when user database was LDAP driven. At that time there were rumors that FreeBSD needs LDAP user-database support, but with the glitches of net/nss_ldap, it seemed that there's no ready-to-implement solution at that time. Things changed completely with net/nss-pam-ldapd. Haven't had any negative experiences with single-LDAP backend networks. Haven't had big environments yet either, but I think it's time to think about base-LDAP-support again. net/nss-pam-ldapd is GPL licensed, so I guess it won't get into base, but it was a great template, is it? -Harry signature.asc Description: OpenPGP digital signature
Re: nscd not caching
On Aug 18, 2014, at 10:42 AM, Eggert, Lars wrote: > Hi, > > On 2014-8-17, at 18:10, Adam McDougall wrote: >> We were using +: type entries in the local password and group >> tables and I believe we used an unmodified /etc/nsswitch.conf (excluding >> cache lines while testing nscd): > > I tried that setup too, and it doesn't seem to be caching any NIS lookups > either. > > The current NIS server is 25ms away, which is a pain. I'm trying to get a > local slave set up, which will make the need for nscd go away, but it would > sure be nice if it worked in the meantime. > >> At our site, we never had enough load to outright require nscd on >> FreeBSD, although there were some areas where caching had a usability >> benefit. > > Load is not an issue, latency is (see above). I know that this a bit late but have you ever considered Hesiod? it uses DNS/txt. we have been using it since the days when BSDi had no NIS support and haven’t seen a ypserver not responding since :-) danny ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: nscd not caching
Eggert, Lars wrote this message on Mon, Aug 18, 2014 at 07:42 +: > The current NIS server is 25ms away, which is a pain. I'm trying to get a > local slave set up, which will make the need for nscd go away, but it would > sure be nice if it worked in the meantime. Why not run a local slave on your server? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: nscd not caching
Am 17.08.2014 um 18:10 schrieb Adam McDougall: > On 08/17/2014 09:09, Eggert, Lars wrote: >> Nobody using nscd? Really? > > I would test for you, but we retired our NIS infrastructure at least a > year ago. I did have it working on a test client at some point, but I > didn't push it into production because I found a couple issues (below). [...] > The two main problems I recall were nscd making java crash, and nscd > holding on to negative cache lookups too long, causing failures while > installing ports that depend on adding users/groups for a following file > permission change. I can't remember if the latter issue was fixed at > some point. I also can't remember if I was receiving perfectly accurate > results from the cache either. I added the "negative-confidence-threshold" option to nscd, a few years ago. If set to a number > 1 (the default), then that number of failures are required to cause a negative cache entry. Setting this value to 3 should allow for 2 probes for the presence of a UID or username, before the cache returns a failure without bothering to re-check the source. The value should be low enough to prevent flooding of a remote source with requests, if an entry really does not exist. The default was left unchanged - you need to increase the value to see any effect of this threshold. 3 might be a reasonable default for the user database. But I never bothered to suggest and discuss an increased default value on the mail-lists ... [...] > I dabbled with nscd a bit after we switched from NIS to LDAP. I think I > recall lookups being slightly slower WITH the cache, plus I would get > some duplicated group entries returned on all but the first getent > group. The short version is we in no way seem to benefit or require a > cache of LDAP with our site size, so I'm just not using nscd. I didn't > make bug reports for these issues, I had to prioritize towards more > pressing issues. I'm trying to do better about reporting bugs. I also found that there were glitches, when I tested the extension to cache only the nth negative reply. The code is not easy to read and change (IMHO), and I did not succeed when I tried to reproduce and debug these glitches. Regards, STefan ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: nscd not caching
Hi, On 2014-8-17, at 18:10, Adam McDougall wrote: > We were using +: type entries in the local password and group > tables and I believe we used an unmodified /etc/nsswitch.conf (excluding > cache lines while testing nscd): I tried that setup too, and it doesn't seem to be caching any NIS lookups either. The current NIS server is 25ms away, which is a pain. I'm trying to get a local slave set up, which will make the need for nscd go away, but it would sure be nice if it worked in the meantime. > At our site, we never had enough load to outright require nscd on > FreeBSD, although there were some areas where caching had a usability > benefit. Load is not an issue, latency is (see above). > top was slow to open since it would load the whole passwd > table first, but top -u was a workaround. Right, I see that issue too. > As a workaround until we retired NIS, I wrote a hack of a script to > merge NIS groups into my local /etc/group files periodically from cron. > Aside from bugs in my script, that worked well. I may end up doing this, too. Given all this, maybe it's time to retire nscd? Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: nscd not caching
On Sunday 17 August 2014 15:22:02 O. Hartmann wrote: > Am Sun, 17 Aug 2014 13:09:10 + > > "Eggert, Lars" schrieb: > > Nobody using nscd? Really? > > I can only speak for myself and I stopped using nscd since the support is > crap. > > A while ago (t > 1 1/2 years) I realised within a OpenLDAP environment, that > when nscd is running, sometimes the system simple "forgets" about root - > this is painful while installing/updating ports and getting interrupted > with a weird error "unknown user root". > > nscd is supposed to be used in large environments where the cost for a user > lookup, like in OpenLDAP, is worse. But obviously FreeBSD isn't used in > that large environments with OpenLDAP and I'm wondering what the purpose of > nscd is. The other problem is that net/nss_ldap and security/pam_ldap have kind of been left behind for performance and robustness. People who use this seriousy tend to discover net/nss-pam-ldapd fairly quickly which has its own caching proxy and eliminates the need for nscd. With nss_ldap and pam_ldap, the openldap client libraries and startup cost is linked into every single binary that uses it. WIth pam-nss-ldapd, it's a simple unix domain socket (with peercred authentication) and no heavy-weight libraries in the consumers. The proxy on the other end of the socket keeps a ldap connection open (with an idle timeout). The whole thing was vastly more robust and efficient. At least that's what we found in the freebsd.org cluster. nss-pam-ldapd was two or three orders of magnitude more usable and got rid of nscd in the process. For us, nscd "worked", but it just didn't save much effort because it was a per-uid cache. ie: if "jkh" had just caused a ldap search, and "peter" repeated it, it had to be done again from scratch. The downside for nss-pam-ldapd was that it uses a non-extensible wire protocol and didn't have room for bsd-style login classes. > > On 2014-8-14, at 13:26, Eggert, Lars wrote: > > > [Resending to current@, since I can't get it to work on -CURRENT > > > either.] > > > > > > Hi, > > > > > > anyone have an idea why nscd would not be caching NIS lookups? > > > > > > My nsswitch.conf looks as follows: > > > > > > group: cache files nis > > > hosts: cache files dns > > > networks: cache files > > > passwd: cache files nis > > > shells: files > > > services: cache files nis > > > protocols: cache files > > > rpc: cache files > > > > > > nisdomain is set and ypbind is started, and I see lots of NIS traffic > > > going in and out. > > > > > > But nothing is cached; running nscd with -t just prints this and then > > > then nothing, ever: > > > > > > M1 from main: successfully daemonized > > > M1 from main: request agents registered successfully > > > M2 from cache: cache was successfully initialized > > > M2 from runtime environment: using socket /var/run/nscd > > > M2 from runtime environment: successfully initialized > > > M1 from main: thread #0 was successfully created > > > M1 from main: thread #1 was successfully created > > > M1 from main: thread #2 was successfully created > > > M1 from main: thread #3 was successfully created > > > M1 from main: thread #4 was successfully created > > > M1 from main: thread #5 was successfully created > > > M1 from main: thread #6 was successfully created > > > M1 from main: thread #7 was successfully created > > > > > > Lars -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV UTF-8: for when a ' or ... just won\342\200\231t do\342\200\246 signature.asc Description: This is a digitally signed message part.
Re: nscd not caching
On 08/17/2014 09:09, Eggert, Lars wrote: > Nobody using nscd? Really? > I would test for you, but we retired our NIS infrastructure at least a year ago. I did have it working on a test client at some point, but I didn't push it into production because I found a couple issues (below). We were using +: type entries in the local password and group tables and I believe we used an unmodified /etc/nsswitch.conf (excluding cache lines while testing nscd): group: compat group_compat: nis hosts: files dns networks: files passwd: compat passwd_compat: nis shells: files services: compat services_compat: nis protocols: files rpc: files The two main problems I recall were nscd making java crash, and nscd holding on to negative cache lookups too long, causing failures while installing ports that depend on adding users/groups for a following file permission change. I can't remember if the latter issue was fixed at some point. I also can't remember if I was receiving perfectly accurate results from the cache either. At our site, we never had enough load to outright require nscd on FreeBSD, although there were some areas where caching had a usability benefit. top was slow to open since it would load the whole passwd table first, but top -u was a workaround. Our Samba servers allowed users to connect a few seconds quicker if it didn't have to pull down the entire group table to check group membership of the connecting user. As a workaround until we retired NIS, I wrote a hack of a script to merge NIS groups into my local /etc/group files periodically from cron. Aside from bugs in my script, that worked well. I dabbled with nscd a bit after we switched from NIS to LDAP. I think I recall lookups being slightly slower WITH the cache, plus I would get some duplicated group entries returned on all but the first getent group. The short version is we in no way seem to benefit or require a cache of LDAP with our site size, so I'm just not using nscd. I didn't make bug reports for these issues, I had to prioritize towards more pressing issues. I'm trying to do better about reporting bugs. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: nscd not caching
Am Sun, 17 Aug 2014 13:09:10 + "Eggert, Lars" schrieb: > Nobody using nscd? Really? I can only speak for myself and I stopped using nscd since the support is crap. A while ago (t > 1 1/2 years) I realised within a OpenLDAP environment, that when nscd is running, sometimes the system simple "forgets" about root - this is painful while installing/updating ports and getting interrupted with a weird error "unknown user root". nscd is supposed to be used in large environments where the cost for a user lookup, like in OpenLDAP, is worse. But obviously FreeBSD isn't used in that large environments with OpenLDAP and I'm wondering what the purpose of nscd is. > > On 2014-8-14, at 13:26, Eggert, Lars wrote: > > > [Resending to current@, since I can't get it to work on -CURRENT either.] > > > > Hi, > > > > anyone have an idea why nscd would not be caching NIS lookups? > > > > My nsswitch.conf looks as follows: > > > > group: cache files nis > > hosts: cache files dns > > networks: cache files > > passwd: cache files nis > > shells: files > > services: cache files nis > > protocols: cache files > > rpc: cache files > > > > nisdomain is set and ypbind is started, and I see lots of NIS traffic going > > in and > > out. > > > > But nothing is cached; running nscd with -t just prints this and then then > > nothing, > > ever: > > > > M1 from main: successfully daemonized > > M1 from main: request agents registered successfully > > M2 from cache: cache was successfully initialized > > M2 from runtime environment: using socket /var/run/nscd > > M2 from runtime environment: successfully initialized > > M1 from main: thread #0 was successfully created > > M1 from main: thread #1 was successfully created > > M1 from main: thread #2 was successfully created > > M1 from main: thread #3 was successfully created > > M1 from main: thread #4 was successfully created > > M1 from main: thread #5 was successfully created > > M1 from main: thread #6 was successfully created > > M1 from main: thread #7 was successfully created > > > > Lars > > > signature.asc Description: PGP signature
Re: nscd not caching
Nobody using nscd? Really? On 2014-8-14, at 13:26, Eggert, Lars wrote: > [Resending to current@, since I can't get it to work on -CURRENT either.] > > Hi, > > anyone have an idea why nscd would not be caching NIS lookups? > > My nsswitch.conf looks as follows: > > group: cache files nis > hosts: cache files dns > networks: cache files > passwd: cache files nis > shells: files > services: cache files nis > protocols: cache files > rpc: cache files > > nisdomain is set and ypbind is started, and I see lots of NIS traffic going > in and out. > > But nothing is cached; running nscd with -t just prints this and then then > nothing, ever: > > M1 from main: successfully daemonized > M1 from main: request agents registered successfully > M2 from cache: cache was successfully initialized > M2 from runtime environment: using socket /var/run/nscd > M2 from runtime environment: successfully initialized > M1 from main: thread #0 was successfully created > M1 from main: thread #1 was successfully created > M1 from main: thread #2 was successfully created > M1 from main: thread #3 was successfully created > M1 from main: thread #4 was successfully created > M1 from main: thread #5 was successfully created > M1 from main: thread #6 was successfully created > M1 from main: thread #7 was successfully created > > Lars > signature.asc Description: Message signed with OpenPGP using GPGMail
nscd not caching
[Resending to current@, since I can't get it to work on -CURRENT either.] Hi, anyone have an idea why nscd would not be caching NIS lookups? My nsswitch.conf looks as follows: group: cache files nis hosts: cache files dns networks: cache files passwd: cache files nis shells: files services: cache files nis protocols: cache files rpc: cache files nisdomain is set and ypbind is started, and I see lots of NIS traffic going in and out. But nothing is cached; running nscd with -t just prints this and then then nothing, ever: M1 from main: successfully daemonized M1 from main: request agents registered successfully M2 from cache: cache was successfully initialized M2 from runtime environment: using socket /var/run/nscd M2 from runtime environment: successfully initialized M1 from main: thread #0 was successfully created M1 from main: thread #1 was successfully created M1 from main: thread #2 was successfully created M1 from main: thread #3 was successfully created M1 from main: thread #4 was successfully created M1 from main: thread #5 was successfully created M1 from main: thread #6 was successfully created M1 from main: thread #7 was successfully created Lars signature.asc Description: Message signed with OpenPGP using GPGMail
