If "setpag" is failing after 16 calls, then yes, I think I know
at least roughly what the problem is, and how to fix it.
I ran into this problem with netatalk, but it sounds like
the identical problem to what's happening in xdm.
Under AIX, permissions are a *lot* richer than with Unix.
Having "root" rights really means having a whole complex of
rights. At any given time, you may have only a subset of your
potential rights active - there are actually 4 sets of
privileges - PRIV_EFFECTIVE, PRIV_INHERITED, PRIV_BEQUEATH,
and PRIV_MAXIMUM. Process credentials also work
a bit oddly: groups are actually stored in a structure
that is shared between the parent and the child process;
each time a process changes the structure it's supposed
to make a copy (decrementing reference counts & possibly
freeing the old copy). The "16" is actually NGROUPS/2, and
that number happens because "setpag" uses 2 groups.
Apparently, under certain circumstances, it's possible
to have a "setuid ROOT" child process that doesn't have
all the rights it should, so setpag apparently doesn't
manage to duplicate the group structure. It can still
add entries, so it starts eating group entries, 2 at a time,
out of the parent processes's group structure, until it
runs out of slots.
A very crude approach is just to put this in:
setuid(0)
just before the setpag(). The setuid apparently assures 2 things: the
process has all the rights it should, and it has its own copy of the
process credentials structure. This was "The Kludge" I used
in netatalk for a long while. For xdm this may be perfectly
adequate. Necessarily this does mean it should still be
running as root when it does the setpag.
A more finessed approach is to use setuidx instead of
setuid (the setuidx avoids the privileges implications
of setuid), and to use getpriv/setpriv to manipulate
the privileges directly. This is what I do today in
netatalk. The whole approach is too complicated
to completely describe here, but I can mail you a copy of
the relevant file (auth.c). Or if you're interested in
looking at all of netatalk, you can contact the folks at
citi ([EMAIL PROTECTED]).
-Marcus Watts
