Bug#717235: proftpd requests the whole passwd database at each login

[Hilmar Preuße]

tags 717235 + pending
stop

Am 25.09.2016 um 18:21 schrieb Arthur de Jong:

On Tue, 2016-09-20 at 14:37 +0200, Hilmar Preuße wrote:


Hi Arthur,



An alternative is to do setpwent()/getpwent()/setpwent() which should
abort the started search without wasting too much time trying to
search LDAP entries in our environment.


...and this is exactly what has been implemented in upstream. ;-)


From a quick look that seems to be the implementation. I haven't done
any testing though but this should work in any case.

Thanks for confirmation! Patch is in git, will be in next upload. Due to 
#820984 (still open) I can't give a time frame for upload.



I still question the usefulness of setpwent()/getpwent() in the first
place as there is no guarantee that any resources opened for setpwent()
are used for other getpw{nam,uid}() calls.



Hilmar
--
http://www.hilmar-preusse.de.vu/   #206401 http://counter.li.org

Bug#717235: proftpd requests the whole passwd database at each login

[Arthur de Jong]

On Tue, 2016-09-20 at 14:37 +0200, Hilmar Preuße wrote:
> 
> An alternative is to do setpwent()/getpwent()/setpwent() which should
> abort the started search without wasting too much time trying to
> search LDAP entries in our environment.
> 
> 
> ...and this is exactly what has been implemented in upstream. ;-)

From a quick look that seems to be the implementation. I haven't done
any testing though but this should work in any case.

I still question the usefulness of setpwent()/getpwent() in the first
place as there is no guarantee that any resources opened for setpwent()
are used for other getpw{nam,uid}() calls.

-- 
-- arthur - adej...@debian.org - https://people.debian.org/~adejong --


signature.asc
Description: This is a digitally signed message part

Bug#717235: proftpd requests the whole passwd database at each login

[Hilmar Preuße]

Am 19.09.2016 um 18:09 schrieb Marco d'Itri:

On Sep 16, Hilmar Preusse  wrote:


Hi Mrco


  /* Per Debian bug report:
   *   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717235
   * we might want to do another set{pw,gr}ent(), to play better with
   * some NSS modules.
   */
  pr_auth_setpwent(p);
  pr_auth_setgrent(p);
}

I guess I can simply replace the ensure_open_passwd() function in
1.3.5 by than one, right? Please confirm, patch will be in next
upload then.


Arthur suggested a different fix: I am not sure about what effect
a setpwent() call would have on nslcd and if it would fix the issue of
the client process requesting the whole database.


https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717235#39


An alternative is to do setpwent()/getpwent()/setpwent() which should 
abort the started search without wasting too much time trying to search 
LDAP entries in our environment.



...and this is exactly what has been implemented in upstream. ;-)

Hilmar
--
http://www.hilmar-preusse.de.vu/   #206401 http://counter.li.org

Bug#717235: proftpd requests the whole passwd database at each login

[Marco d'Itri]

On Sep 16, Hilmar Preusse  wrote:

>   /* Per Debian bug report:
>*   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717235
>* we might want to do another set{pw,gr}ent(), to play better with
>* some NSS modules.
>*/
>   pr_auth_setpwent(p);
>   pr_auth_setgrent(p);
> }
> 
> I guess I can simply replace the ensure_open_passwd() function in
> 1.3.5 by than one, right? Please confirm, patch will be in next
> upload then.
Arthur suggested a different fix: I am not sure about what effect 
a setpwent() call would have on nslcd and if it would fix the issue of 
the client process requesting the whole database.
Arthur?

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Bug#717235: proftpd requests the whole passwd database at each login

[Hilmar Preuße]

tags 717235 + fixed-upstream
# in 1.3.6rc2
stop

Bug#717235: proftpd requests the whole passwd database at each login

[Hilmar Preusse]

tags 717235 + fixed-upstream
# in 1.3.6rc2
stop

On 08.01.16 Marco d'Itri (m...@linux.it) wrote:
> On Sep 17, Marco d'Itri  wrote:

Hi all,

> > > After I switched from libnss-ldap to libnss-ldapd I noticed that at 
> > > every login proftpd requests the whole content of the passwd database 
> > > (i.e. like running "getent passwd).
> > > This is evident from the nslcd debugging log ("passwd(all)").
> > Are there any news about fixing this?
> > This bug prevents using proftpd on systems with a large users database.
> 
> Since there has been no reaction from the maintainer in 2.5 years 
> I intend to NMU proftpd in the next future and apply the patch in this 
> bug. (And also fix #804322).
> 
Current implementation in 1.3.6rc2 looks like this:

static void ensure_open_passwd(pool *p) {
  /* Make sure pass/group is open. */
  pr_auth_setpwent(p);
  pr_auth_setgrent(p);

  /* On some unices the following is necessary to ensure the files
   * are open (BSDI 3.1)
   */
  pr_auth_getpwent(p);
  pr_auth_getgrent(p);

  /* Per Debian bug report:
   *   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717235
   * we might want to do another set{pw,gr}ent(), to play better with
   * some NSS modules.
   */
  pr_auth_setpwent(p);
  pr_auth_setgrent(p);
}

I guess I can simply replace the ensure_open_passwd() function in
1.3.5 by than one, right? Please confirm, patch will be in next
upload then.

H.
-- 
sigmentation fault


signature.asc
Description: PGP signature

Bug#717235: proftpd requests the whole passwd database at each login

[Marco d'Itri]

On Sep 17, Marco d'Itri  wrote:

> > After I switched from libnss-ldap to libnss-ldapd I noticed that at 
> > every login proftpd requests the whole content of the passwd database 
> > (i.e. like running "getent passwd).
> > This is evident from the nslcd debugging log ("passwd(all)").
> Are there any news about fixing this?
> This bug prevents using proftpd on systems with a large users database.
Since there has been no reaction from the maintainer in 2.5 years 
I intend to NMU proftpd in the next future and apply the patch in this 
bug. (And also fix #804322).

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Bug#717235: proftpd requests the whole passwd database at each login

[PICCORO McKAY Lenz]

ping? any news? this made unusable the current debian stable brand respect
the oldstable (that works perfectly) in real deployment infraestructures
with large users database..

i dont have a server for toy in my house, server software are for deploy in
large environments..

Lenz McKAY Gerardo (PICCORO)
http://qgqlochekone.blogspot.com

Bug#717235: proftpd requests the whole passwd database at each login

[Marco d'Itri]

On Jul 18, Marco d'Itri m...@linux.it wrote:

 After I switched from libnss-ldap to libnss-ldapd I noticed that at 
 every login proftpd requests the whole content of the passwd database 
 (i.e. like running getent passwd).
 This is evident from the nslcd debugging log (passwd(all)).
Are there any news about fixing this?
This bug prevents using proftpd on systems with a large users database.

-- 
ciao,
Marco


signature.asc
Description: Digital signature

Bug#717235: proftpd requests the whole passwd database at each login

[Arthur de Jong]

I've had a look at the source of proftpd and it could be that using the
PersistentPasswd option could help in this case, however LDAP users
would not show up so it may not be an option for your environment.

Digging further I tracked the probable culprit to the
ensure_open_passwd() function in mod_auth.c which does a dummy
setpwent()/getpwent() call without endpwent() to try to ensure that the
passwd file is open.

This function is called from setup_env() which is called from
auth_pass() which seems to be called for every authentication attempt.

Attached is a patch that comments out the getpwent() part from
ensure_open_passwd(). This should be sufficient for libnss-ldapd because
it only starts a search on requesting the first result.

Whether the original implementation really works as intended is
questionable because it is not guaranteed that getpwnam() would use the
same file that is used for getpwent(). The original goal of the function
seems to be to be able to do name lookups after going into a chroot
jail.

An alternative is to do setpwent()/getpwent()/setpwent() which should
abort the started search without wasting too much time trying to search
LDAP entries in our environment.

-- 
-- arthur - adej...@debian.org - http://people.debian.org/~adejong --
Description: Improve handling of getpwent calls
 This removes a dummy getpwent() call that causes problems with
 libnss-ldapd and large LDAP environments.
Author: Arthur de Jong adej...@debian.org
Bug-Debian: http://bugs.debian.org/717235

--- a/modules/mod_auth.c
+++ b/modules/mod_auth.c
@@ -830,8 +830,11 @@ static void ensure_open_passwd(pool *p)
   /* On some unices the following is necessary to ensure the files
* are open.  (BSDI 3.1)
*/
+/* Disabled this on Debian because it causes problems in large environments,
+   see http://bugs.debian.org/717235
   pr_auth_getpwent(p);
   pr_auth_getgrent(p);
+*/
 }
 
 /* Next function (the biggie) handles all authentication, setting


signature.asc
Description: This is a digitally signed message part

Bug#717235: proftpd requests the whole passwd database at each login

[Arthur de Jong]

On Thu, 2013-07-18 at 11:48 +0200, Marco d'Itri wrote:
 While this can be overlooked on normal systems, it obviously has huge 
 effects on the LDAP server performances, since my passwd database 
 contains hundreds of thousands of entries.
 
 I do not know why the issue was not evident with libnss-ldap too, maybe 
 Arthur knows more about this part.

Are you saying there is a significant performance difference when
running getent passwd (or running proftpd) in your environment between
libnss-ldap and libnss-ldapd?

If so, please open a bugreport against nss-pam-ldapd.

I can't think of a reason for proftpd to do anything different between
libnss-ldap and libnss-ldapd as they both implement the exact same API.

-- 
-- arthur - adej...@debian.org - http://people.debian.org/~adejong --


signature.asc
Description: This is a digitally signed message part

Bug#717235: proftpd requests the whole passwd database at each login

[Marco d'Itri]

On Jul 19, Arthur de Jong adej...@debian.org wrote:

 Are you saying there is a significant performance difference when
 running getent passwd (or running proftpd) in your environment between
 libnss-ldap and libnss-ldapd?
Yes: getent passwd works on both systems, but when I switch from 
libnss-ldap to libnss-ldapd proftpd generates tens of Mbps of LDAP 
traffic with these all queries.

I have a theory, but I have not verified it by looking at the code: 
I can see in the nslcd debug log that passwd(all) is requested, but 
then only a few lines are listed in the log (and IIRC they are followed 
by an error which suggests that the client stopped requesting data).
So I wonder if the problem is that:
- proftpd requests passwd(all)
- but it only looks at the first few results and then calls endpwent(3) 
  or something like this
- libnss-ldap then would immediately stop requesting records from the 
  LDAP server
- but libnss-ldapd uses nslcd which is persistent, so nslcd would still 
  receive all data even if the client does not care anymore

Does this look reasonable to you?
If it is true then I do not think that it would be a libnss-ldapd bug.

-- 
ciao,
Maro


signature.asc
Description: Digital signature

Bug#717235: proftpd requests the whole passwd database at each login

[Arthur de Jong]

On Fri, 2013-07-19 at 12:53 +0200, Marco d'Itri wrote:
 I have a theory, but I have not verified it by looking at the code: 
 I can see in the nslcd debug log that passwd(all) is requested, but 
 then only a few lines are listed in the log (and IIRC they are followed 
 by an error which suggests that the client stopped requesting data).
 So I wonder if the problem is that:
 - proftpd requests passwd(all)
 - but it only looks at the first few results and then calls endpwent(3) 
   or something like this
 - libnss-ldap then would immediately stop requesting records from the 
   LDAP server
 - but libnss-ldapd uses nslcd which is persistent, so nslcd would still 
   receive all data even if the client does not care anymore
 
 Does this look reasonable to you?
 If it is true then I do not think that it would be a libnss-ldapd bug.

That would indeed be a difference between libnss-ldap and
libnss-ldapd/nslcd. nslcd has by default 5 threads that handle requests
and pass the results back to the NSS module. To achieve reasonable
performance and not tie up the threads too long, the communication
between nslcd and the NSS module is buffered (on both ends) with
reasonably large buffers.

This could result in considerable more results being requested from the
LDAP server with libnss-ldapd/nslcd than with libnss-ldap.

-- 
-- arthur - adej...@debian.org - http://people.debian.org/~adejong --


signature.asc
Description: This is a digitally signed message part

Bug#717235: proftpd requests the whole passwd database at each login

[Marco d'Itri]

Package: proftpd
Version: 1.3.4a-4+nmu1
Severity: important

After I switched from libnss-ldap to libnss-ldapd I noticed that at 
every login proftpd requests the whole content of the passwd database 
(i.e. like running getent passwd).
This is evident from the nslcd debugging log (passwd(all)).

While this can be overlooked on normal systems, it obviously has huge 
effects on the LDAP server performances, since my passwd database 
contains hundreds of thousands of entries.

I do not know why the issue was not evident with libnss-ldap too, maybe 
Arthur knows more about this part.

-- 
ciao,
Marco


signature.asc
Description: Digital signature

Bug#717235: proftpd requests the whole passwd database at each login

[Andrei POPESCU]

Control: reassign -1 src:proftpd-dfsg 1.3.4a-4+nmu1

On Jo, 18 iul 13, 11:48:23, Marco d'Itri wrote:
 Package: proftpd
 Version: 1.3.4a-4+nmu1

This binary package doesn't exist and I can't tell which one you meant, 
reassigning to the source package instead.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser
Offtopic discussions among Debian users and developers:
http://lists.alioth.debian.org/mailman/listinfo/d-community-offtopic


signature.asc
Description: Digital signature

Bug#717235: proftpd requests the whole passwd database at each login

[Marco d'Itri]

On Jul 18, Andrei POPESCU andreimpope...@gmail.com wrote:

 This binary package doesn't exist and I can't tell which one you meant, 
 reassigning to the source package instead.
Oops. I am using proftpd-base.

-- 
ciao,
Marco


signature.asc
Description: Digital signature