Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marco d'Itri]

On Apr 07, Bernd Zeimetz  wrote:

> There are more than enough ways to keep the entries based on dns
> records in your l3 firewalls uptodate, I can't see how this should
> warrant to keep yet another patch Jan^WMarco.
Not for the form *.domain.tld.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Bernd Zeimetz]

On Tue, 2024-04-02 at 12:04 +0200, Marco d'Itri wrote:
> On Apr 02, Colin Watson  wrote:
> 
> > At the time, denyhosts was popular, but it was removed from Debian
> > several years ago.  I remember that, when I dealt with that on my
> > own
> > systems, fail2ban seemed like the obvious replacement, and my
> > impression
> > is that it's pretty widely used nowadays; it's very pluggable but
> > it
> > normally works by adding firewall rules.  Are there any similar
> > popular
> > systems left that rely on editing /etc/hosts.deny?
> Yes, people. I object to removing TCP wrappers support since the
> patch 
> is tiny and it supports use cases like DNS-based ACLs which cannot be
> supported by L3 firewalls.
> 

There are more than enough ways to keep the entries based on dns
records in your l3 firewalls uptodate, I can't see how this should
warrant to keep yet another patch Jan^WMarco.

-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Thu, Apr 04, 2024 at 06:42:08PM -0300, Henrique de Moraes Holschuh wrote:
> If libwrap is bringing in complex libs, maybe we could reduce the
> attack surface on libwrap itself?  It would be nice to have a variant
> that only links to the libc and that's it...

Yeah, that's https://bugs.debian.org/1068311 which I linked to elsewhere
in this thread.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Henrique de Moraes Holschuh]

On Tue, Apr 2, 2024, at 07:04, Marco d'Itri wrote:
> On Apr 02, Colin Watson  wrote:
>
>> At the time, denyhosts was popular, but it was removed from Debian
>> several years ago.  I remember that, when I dealt with that on my own
>> systems, fail2ban seemed like the obvious replacement, and my impression
>> is that it's pretty widely used nowadays; it's very pluggable but it
>> normally works by adding firewall rules.  Are there any similar popular
>> systems left that rely on editing /etc/hosts.deny?
> Yes, people. I object to removing TCP wrappers support since the patch 
> is tiny and it supports use cases like DNS-based ACLs which cannot be 
> supported by L3 firewalls.

If libwrap is bringing in complex libs, maybe we could reduce the attack 
surface on libwrap itself?  It would be nice to have a variant that only links 
to the libc and that's it...

And that benefits everything that links to TCP wrappers...

I also like to have the (old-school) standard extra layer of protection that 
libwrap can provide. I'd like to find a way to keep it useful for sshd.

-- 
  Henrique de Moraes Holschuh 

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marc Haber]

On Thu, 4 Apr 2024 13:25:04 +0200, Stephan Seitz
 wrote:
>Am Di, Apr 02, 2024 at 13:30:43 +0200 schrieb Marc Haber:
>>from being vulnerable to the current xz-based attack. Just having to
>>dump an ALL: ALL into /etc/hosts.deny is vastly easier than having to
>>maintain a packet filter.
>
>Stupid question, but if you put „ALL: ALL” into hosts.deny, couldn’t you 
>stop the ssh daemon instead? ALL: ALL will block your ssh access, so it 
>doesn’t matter if the daemon is running or not.

Of course there are sshd: lines in hosts.allow for "my" networks.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Russ Allbery]

Florian Lohoff  writes:

> These times have long gone and tcp wrapper as a security mechanism has
> lost its reliability, this is why people started moving away from tcp
> wrapper (which i think is a shame)

> I personally moved to nftables which is nearly as simple once you get
> your muscle memory set. If ssh is your only candidate of network service
> you could also use match statements in /etc/ssh/sshd_config.d/.

For what it's worth, I have iptables (I know, it's nftables under the hood
now, but I'm still using the iptables syntax because the number of hours
in each day is annoyingly low) on every system I run and I still use TCP
wrappers for ssh restrictions for one host.  That's because I have users
who use various ISPs, and for some of those ISPs, DNS-based restrictions
are less maintenance work than playing whack-a-mole with their
ever-changing IP blocks.

Yes, yes, I know this isn't actually secure, etc., but that's fine, I'm
not using it as a primary security measure.  I'm using it to narrow the
number of hosts on the Internet that can exploit an sshd vulnerability,
and to reduce the amount of annoying automated exploit attempts I get.
(Exactly the kind of thing that helps mildly against situations like the
xz backdoor.)

That said, the point that I could switch over to Match blocks in the sshd
configuration is well-taken, and not wanting to take an hour to rewrite my
rules in a different configuration format is probably not a good enough
reason to keep a dependency in a security-critical, network-exposed
service.  I'm mildly grumbly becuase it's yet another thing I have to
change just to keep things from breaking, but such is life.

-- 
Russ Allbery (r...@debian.org)  

ufw (was Re: Debian openssh option review: considering splitting out GSS-API key exchange)

[Holger Levsen]

On Thu, Apr 04, 2024 at 01:32:11PM +0200, Marc Haber wrote:
> So you have dedicated packet filters on every machine you run, even if
> sshd is the only network-facing service?

on most machines and it was as simple as doing:

apt install ufw
ufw allow ssh
ufw enable

voila, done. rules configured like above end up in /etc/ufw/user.rules and
user6.rules. quite simple, quite nice.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Kinda weird that we’re all gonna experience climate change as a series of
short, apocalyptic videos until eventually it’s your phone that’s recording.
(@shocks)


signature.asc
Description: PGP signature

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marc Haber]

On Thu, 4 Apr 2024 13:03:50 +0200, Florian Lohoff  wrote:
>I personally moved to nftables which is nearly as simple once you get
>your muscle memory set.

So you have dedicated packet filters on every machine you run, even if
sshd is the only network-facing service?

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Stephan Seitz]

Am Di, Apr 02, 2024 at 13:30:43 +0200 schrieb Marc Haber:

from being vulnerable to the current xz-based attack. Just having to
dump an ALL: ALL into /etc/hosts.deny is vastly easier than having to
maintain a packet filter.


Stupid question, but if you put „ALL: ALL” into hosts.deny, couldn’t you 
stop the ssh daemon instead? ALL: ALL will block your ssh access, so it 
doesn’t matter if the daemon is running or not.


Stephan

--
|If your life was a horse, you'd have to shoot it.|

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Florian Lohoff]

On Tue, Apr 02, 2024 at 01:30:43PM +0200, Marc Haber wrote:
> On Tue, 2 Apr 2024 01:30:10 +0100, Colin Watson 
> wrote:
> >We carry a patch to restore support for TCP wrappers, which was dropped
> >in OpenSSH 6.7 (October 2014); see
> >https://lists.mindrot.org/pipermail/openssh-unix-dev/2014-April/032497.html
> >and thread.  That wasn't long before the Debian 8 (jessie) freeze, and
> >so I patched it back in "temporarily", but then I dropped the ball on
> >organizing a proper transition. 
> 
> Please don't drop the mechanism that saved my¹ unstable installations
> from being vulnerable to the current xz-based attack. Just having to
> dump an ALL: ALL into /etc/hosts.deny is vastly easier than having to
> maintain a packet filter.
> 
> Greetings
> Marc
> 
> ¹ and probably thousands others

In the good old days we relied on any network facing service to be
linked to tcp wrappers so a single line would secure your system against
the network with all the possible intruders. This is how i worked for
decades.

These times have long gone and tcp wrapper as a security mechanism has
lost its reliability, this is why people started moving away from tcp
wrapper (which i think is a shame)

I personally moved to nftables which is nearly as simple once you get
your muscle memory set. If ssh is your only candidate of network service
you could also use match statements in /etc/ssh/sshd_config.d/.

So - i am okay with removing the libwrap dependency (not happy)

Flo
-- 
Florian Lohoff f...@zz.de
  Any sufficiently advanced technology is indistinguishable from magic.


signature.asc
Description: PGP signature

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Wed, Apr 03, 2024 at 04:01:34PM -0400, Michael Stone wrote:
> To speed things up for those who really want it, perhaps make
> openssh-client/server dependency-only packages on
> openssh-client/server-nogss? People can choose the less-compatible version
> for this release if they want to, and the default can change next release.
> Pushing back the ability to install the unpatched version for a few more
> years seems suboptimal.

I worry about churn, and about inviting more bugs to do with moving an
awkward combination of conffiles and non-conffile configuration files
between packages.  But maybe.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Michael Stone]

On Tue, Apr 02, 2024 at 01:30:10AM +0100, Colin Watson wrote:

  * add dependency-only packages called something like
openssh-client-gsskex and openssh-server-gsskex, depending on their
non-gsskex alternatives
  * add NEWS.Debian entry saying that people need to install these
packages if they want to retain GSS-API key exchange support
  * add release note saying the same

* for Debian trixie+1 (or maybe after the next Ubuntu LTS, depending on
  exact timings):

  * add separate openssh-gsskex source package, carrying gssapi.patch
in addition to whatever's in openssh, and whose binary packages
Conflicts/Replaces/Provides the corresponding ones from openssh
  * add some kind of regular CI to warn about openssh-gsskex being out
of date relative to openssh
  * drop gssapi.patch from openssh, except for small patches to
configuration file handling to accept the relevant options with
some kind of informative warning (compare
https://bugs.debian.org/152657)


To speed things up for those who really want it, perhaps make 
openssh-client/server dependency-only packages on 
openssh-client/server-nogss? People can choose the less-compatible 
version for this release if they want to, and the default can change 
next release. Pushing back the ability to install the unpatched version 
for a few more years seems suboptimal.

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Wed, Apr 03, 2024 at 04:38:19PM +0200, Marc Haber wrote:
> On Wed, 03 Apr 2024 14:10:37 +0100, "Jonathan Dowland"
>  wrote:
> >For you and fellow greybeards, perhaps: I'd be surprised if many people
> >younger than us have even heard of tcp wrappers. I don't think the
> >muscle memory of a diminishing set of users is a strong argument,
> >especially given it's a preference rather than a requirement, and
> >alternatives do exist.
> 
> It is possible to have that alternative not present without being
> noticed (for example, a firewall build script failing, but sshd start
> nof failing), whilea security measure built into the very daemon is
> way harder to be accidentally disabled while keeping the daemon
> running.

While I'm still not totally convinced, one possible alternative here is
https://bugs.debian.org/1068311.  That would still mean one more library
than strictly needed (once the GSS-API stuff is split out), but at least
it would be one small library rather than a big linkage chain over 30
times the size.  I could probably justify keeping it in that case.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marc Haber]

On Wed, 03 Apr 2024 14:10:37 +0100, "Jonathan Dowland"
 wrote:
>On Tue Apr 2, 2024 at 12:30 PM BST, Marc Haber wrote:
>> Please don't drop the mechanism that saved my¹ unstable installations
>> from being vulnerable to the current xz-based attack. Just having to
>> dump an ALL: ALL into /etc/hosts.deny is vastly easier than having to
>> maintain a packet filter.
>
>For you and fellow greybeards, perhaps: I'd be surprised if many people
>younger than us have even heard of tcp wrappers. I don't think the
>muscle memory of a diminishing set of users is a strong argument,
>especially given it's a preference rather than a requirement, and
>alternatives do exist.

It is possible to have that alternative not present without being
noticed (for example, a firewall build script failing, but sshd start
nof failing), whilea security measure built into the very daemon is
way harder to be accidentally disabled while keeping the daemon
running.

I have spent weeks if not months of my life building firewalls that
fail to the safe side (have it "all closed" if something fails during
build), lost them all when we got migrated to nft to do its inadequate
tooling, while hosts.deny and hosts.allow is done in seconds even if
you don't have orchestration.

If there are arguments for keeping tcp-wrappers-compatible security in
sshd, it is NOT muscle memory, it is a techincal founded and solid
preference.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Jonathan Dowland]

On Tue Apr 2, 2024 at 12:30 PM BST, Marc Haber wrote:
> Please don't drop the mechanism that saved my¹ unstable installations
> from being vulnerable to the current xz-based attack. Just having to
> dump an ALL: ALL into /etc/hosts.deny is vastly easier than having to
> maintain a packet filter.

For you and fellow greybeards, perhaps: I'd be surprised if many people
younger than us have even heard of tcp wrappers. I don't think the
muscle memory of a diminishing set of users is a strong argument,
especially given it's a preference rather than a requirement, and
alternatives do exist.

-- 
Please do not CC me for listmail.

  Jonathan Dowland
✎j...@debian.org
   https://jmtd.net

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[RL]

Colin Watson  writes:

> GSS-API key exchange
> 

> However, OpenSSH upstream has long rejected it

> All the same, I'm aware that some people now depend on having this
> facility in Debian's main openssh package


> How does this rough plan sound?
>
>  * for Debian trixie (current testing):
>
>* add dependency-only packages called something like
>  openssh-client-gsskex and openssh-server-gsskex, depending on their
>  non-gsskex alternatives
>* add NEWS.Debian entry saying that people need to install these
>  packages if they want to retain GSS-API key exchange support
>* add release note saying the same

happy to help on release-notes.

Think you've got two audiences:

- people who rely on gss, who may be upgrading over ssh and need to know
  how to avoid being locked out (eg: make sure to install gsskex
  recommended packages before reboot?)

- people who dont use gss, and want to remove it asap: as well as
  removing the gsskex packages would they need to edit sshd_config or
  ssh_config etc -- these can currently contain things like
  'GSSAPIAuthentication no' which would (i assume) stop working (and
  cause sshd to not start) once the gss support is removed(?)

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marc Haber]

On Tue, 2 Apr 2024 01:30:10 +0100, Colin Watson 
wrote:
>We carry a patch to restore support for TCP wrappers, which was dropped
>in OpenSSH 6.7 (October 2014); see
>https://lists.mindrot.org/pipermail/openssh-unix-dev/2014-April/032497.html
>and thread.  That wasn't long before the Debian 8 (jessie) freeze, and
>so I patched it back in "temporarily", but then I dropped the ball on
>organizing a proper transition. 

Please don't drop the mechanism that saved my¹ unstable installations
from being vulnerable to the current xz-based attack. Just having to
dump an ALL: ALL into /etc/hosts.deny is vastly easier than having to
maintain a packet filter.

Greetings
Marc

¹ and probably thousands others
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Tue, Apr 02, 2024 at 12:04:26PM +0200, Marco d'Itri wrote:
> Yes, people. I object to removing TCP wrappers support since the patch 
> is tiny and it supports use cases like DNS-based ACLs which cannot be 
> supported by L3 firewalls.

I suspect OpenSSH upstream would also want me to point out that
DNS-based ACLs are supported by Match blocks without needing a separate
library.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marco d'Itri]

On Apr 02, Colin Watson  wrote:

> You could use a drop-in unit to wrap sshd in tcpd, as suggested by the
> Fedora wiki page?  This would avoid exposing sshd's process space to
> libwrap and all the stuff it links to by default.
This would require to switch to socket activation of sshd, which is not 
the default for good reasons.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Tue, Apr 02, 2024 at 12:04:26PM +0200, Marco d'Itri wrote:
> On Apr 02, Colin Watson  wrote:
> > At the time, denyhosts was popular, but it was removed from Debian
> > several years ago.  I remember that, when I dealt with that on my own
> > systems, fail2ban seemed like the obvious replacement, and my impression
> > is that it's pretty widely used nowadays; it's very pluggable but it
> > normally works by adding firewall rules.  Are there any similar popular
> > systems left that rely on editing /etc/hosts.deny?
> 
> Yes, people. I object to removing TCP wrappers support since the patch 
> is tiny and it supports use cases like DNS-based ACLs which cannot be 
> supported by L3 firewalls.

It's not about the size of the patch.

You could use a drop-in unit to wrap sshd in tcpd, as suggested by the
Fedora wiki page?  This would avoid exposing sshd's process space to
libwrap and all the stuff it links to by default.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Christian Göttsche]

On Tue, 2 Apr 2024 at 02:30, Colin Watson  wrote:
>
> [I've CCed openssh-unix-dev for awareness, but set Mail-Followup-To to
> just debian-devel and debian-ssh to avoid potentially spamming them with
> a long discussion.  If you choose to override this then that's your
> call, but please be mindful of upstream's time.]
>
> Following the xz-utils backdoor, I'm reconsidering some choices in
> Debian's OpenSSH packaging.  Please note that significant rearchitecture
> of the upstream code is out of scope for the Debian packaging, so I'm
> going to disregard comments of the form "maybe there should be a module
> loader so all these things can just be plug-ins" or other such blue-sky
> things; from my point of view this is just about configuring things a
> bit more wisely within more or less our current constraints.
>
>
> libsystemd
> ==
>
> This is the obvious thing on everyone's mind right now.  At the time I
> merged that patch, "not NIHing code that's in a perfectly good library"
> seemed like a reasonable trade-off, but we do seem to have ended up on
> the wrong side of history on this one.  There's work in progress to land
> readiness protocol notification upstream without libsystemd (thanks
> Damien and Luca!), and I expect to cherry-pick this into Debian once
> it's agreed, so we'll get rid of that linkage and reduce our patch load
> a bit.
>
> We also have a patch from Ubuntu to support the systemd socket
> activation protocol.  I've rewritten this to avoid using libsystemd, and
> I'll submit it upstream once the readiness notification work is sorted
> out.  But it's not particularly invasive once the libsystemd linkage is
> removed, so it's not the end of the world if this ends up staying in our
> patch queue.
>
>
> GSS-API key exchange
> 
>
> Way back in 2005, I merged the GSS-API key exchange patch into Debian's
> main openssh package (https://bugs.debian.org/275472).  At the time it
> seemed like a sensible overall reduction in maintenance burden (if I
> remember correctly, the openssh-krb5 package often ended up lagging a
> fair bit behind openssh).  While the patch is fairly large, it hasn't
> generally been too hard to forward-port to newer versions of OpenSSH,
> and Fedora carries it too so there's some sharing of work.
>
> However, OpenSSH upstream has long rejected it, mainly on the basis that
> they don't like adding new pre-authentication attack surface, and this
> week seems like a good one to reconsider what patches we're shipping by
> default.  gssapi.patch is the largest patch in our openssh package by an
> order of magnitude, and easily the most intrusive in terms of complexity
> and exposure, so I've somewhat regretted my choice to merge it a few
> times over the years.
>
> All the same, I'm aware that some people now depend on having this
> facility in Debian's main openssh package: I get enough occasional bug
> reports to convince me that it's still in use.  So, if I decide to split
> it back out, I'd want to arrange for a somewhat graceful transition.
> We've had it for nearly 20 years now, so we can take the time to do a
> proper job that at least tries not to leave users in the lurch.
>
> How does this rough plan sound?
>
>  * for Debian trixie (current testing):
>
>* add dependency-only packages called something like
>  openssh-client-gsskex and openssh-server-gsskex, depending on their
>  non-gsskex alternatives
>* add NEWS.Debian entry saying that people need to install these
>  packages if they want to retain GSS-API key exchange support
>* add release note saying the same
>
>  * for Debian trixie+1 (or maybe after the next Ubuntu LTS, depending on
>exact timings):
>
>* add separate openssh-gsskex source package, carrying gssapi.patch
>  in addition to whatever's in openssh, and whose binary packages
>  Conflicts/Replaces/Provides the corresponding ones from openssh
>* add some kind of regular CI to warn about openssh-gsskex being out
>  of date relative to openssh
>* drop gssapi.patch from openssh, except for small patches to
>  configuration file handling to accept the relevant options with
>  some kind of informative warning (compare
>  https://bugs.debian.org/152657)
>
> I guess we should decide whether the separate packages are to be needed
> for GSS-API authentication as well as key exchange, because that affects
> the choice of dependency-only package names in trixie.  If we only split
> out gssapi.patch (for key exchange; sorry about the slightly misleading
> name) but kept --with-kerberos5 (which also controls authentication),
> then we'd significantly reduce our patch load but not sshd's linkage.
>
> I've seen the suggestion of using libgssglue here
> (https://fosstodon.org/@jas/112194876950058188).  That might be a good
> idea and I have no particular objection to it, though I also don't know
> much about it and it would probably be better if an expert did the work.
> 

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Marco d'Itri]

On Apr 02, Colin Watson  wrote:

> At the time, denyhosts was popular, but it was removed from Debian
> several years ago.  I remember that, when I dealt with that on my own
> systems, fail2ban seemed like the obvious replacement, and my impression
> is that it's pretty widely used nowadays; it's very pluggable but it
> normally works by adding firewall rules.  Are there any similar popular
> systems left that rely on editing /etc/hosts.deny?
Yes, people. I object to removing TCP wrappers support since the patch 
is tiny and it supports use cases like DNS-based ACLs which cannot be 
supported by L3 firewalls.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

On Tue, Apr 02, 2024 at 03:27:30AM +0200, Christoph Anton Mitterer wrote:
> Do you think it will be possible to have still only one `ssh`, `scp`,
> etc. command and that will just use extra GSSAPI stuff if installed and
> needed by a certain connection?

It would be technically possible to retain the client parts of the
GSS-API key exchange patch in the default variant.  It would require the
build to be separated into multiple passes, since that patch touches a
number of files shared by the client and the server.

Rather than trying to construct this, though, it would be much simpler
and I think safer to just have a separate openssh-client-gsskeyex
package.  Like today's openssh-client, it would be usable both with and
without GSS-API key exchange.

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Howard Chu]

Damien Miller wrote:
> Another thing we're considering in OpenSSH is changing how we integrate
> with PAM. PAM's API demands loading modules into the authenticating
> process' address space, but obviously we've just been reminded that this
> is risky.

This was a long-standing problem with pam/nss-ldap, which we solved by moving 
all of the
actual libldap invocations to a separate nslcd process, and only communicated 
to it
across a unix domain socket via stubs in the pam/nss modules. Mixing instances 
of libraries
that applications call directly, with instances loaded implicitly by 
system-level mechanisms,
was always a bad idea and caused endless problems, even without malicious 
attackers.

> I think that I would prefer to move to a model where there PAM auth and
> account modules run in a helper process, and only the session module
> runs in the unprivileged post-auth sshd process.

We could probably generalize the stub wrapping that we used for nss/pam-ldapd / 
nslcd to
be a generic interface to a standalone pamd that actually loads the pam 
modules. Should
be a simple job.

> This means that PAM auth/account modules and their transitive library
> dependencies cannot affect the sshd address space. They would still
> likely need to run with privilege, could still fail permissively in
> unwanted situations and might still be able to cause problems directly
> (e.g. opening a reverse shell from the PAM module itself), but they
> would no longer have direct access to the contents of sshd network
> traffic, signatures, etc that are extremely useful in building NOBUS
> (https://en.wikipedia.org/wiki/NOBUS) backdoors like the xv one.
> 
> Where this gets challenging is that some PAM modules make assumptions
> that the auth, account and session modules all run in the same address
> space. These would break until re-architected to pass things explicitly,
> e.g. via environment variables, temp files, etc.

-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Sirius]

In days of yore (Tue, 02 Apr 2024), Colin Watson thus quoth: 
> TCP wrappers
> 

Not used hosts.{allow,deny} for the last 17 years (since I started my
current employment) so I am biased. Honest opinion is that firewall and
fail2ban have pretty much obsoleted TCP wrappers.

> SELinux
> ===
> 
> For the time being my inclination is to leave this be, but I've seen the
> suggestion that pam_selinux is basically all you need
> (https://infosec.exchange/@alwayscurious/112192949171400643), so maybe
> it would be an option to drop --with-selinux in favour of that?  I've
> never used SELinux, so I'd need an expert to weigh on here.

If you need an expert on SELinux, you need Dan Walsh.

I have used SELinux for the last 17 years, from when it was a monolithic
policy to what it is like today in RHEL. SELinux is - as far as I know -
not an issue and have a fail-close rather than fail-open approach. IMHO,
if it is not used and you have the time to spare to drop it, do, otherwise
it should be safe with the status-quo on this.

And should Debian pick SELinux up fully and enable a targeted policy,
well, you will want this anyway.

-- 
Kind regards,

/S

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Russ Allbery]

Christoph Anton Mitterer  writes:

> Actually I think that most sites where I "need"/use GSSAPI... only
> require the ticket for AFS, and do actually allow pubkey auth (but
> right now, one doesn't have AFS access then).

In past discussions of this patch, this has not been the case.  One of the
advantages of GSSAPI key exchange is that you can disable public keys for
all of your hosts and never manage known hosts, instead only using the
system Kerberos keytabs.  Since in a Kerberos environment you have to put
keytabs on every host *anyway*, and that *is* the host's identity in a
Kerberos environment, this reduces the number of key infrastructures you
have to manage by one, which matters to some Kerberos deployments.  This
arguably gives you better security in that specific environment because
keytabs do not rely on leap-of-faith initial authentication; the server is
always properly authenticated, even on first connect.

> Not sure if there's a simple out of the box way to just transfer that
> but without all the other GSSAPI stuff?

If you want your ticket to refresh remotely when you refresh it locally,
which is often needed for Kerberos applications like AFS, you do need key
exchange, since that's the mechanism that allows that to happen.

(I use both GSSAPI and tcpwrappers, so Colin's proposal would mean more
work for me, but given the situation, I'm willing to rework the way that I
use ssh to avoid both going forward.  More features are nice, but I can
see the merits of simplicity here.  But I no longer maintain a large
infrastructure built on Kerberos, so I'm not putting as much weight on the
GSSAPI support as I used to.)

-- 
Russ Allbery (r...@debian.org)  

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Christoph Anton Mitterer]

Hey.

On Tue, 2024-04-02 at 01:30 +0100, Colin Watson wrote:
> All the same, I'm aware that some people now depend on having this
> facility in Debian's main openssh package: I get enough occasional
> bug
> reports to convince me that it's still in use.

Being one of those people, and having even asked for more patches to be
merged (#1053822) O:-) ... no, there was no social engineering
intended,... your approach sounds like a reasonable way to go.

Actually I think that most sites where I "need"/use GSSAPI... only
require the ticket for AFS, and do actually allow pubkey auth (but
right now, one doesn't have AFS access then).
Not sure if there's a simple out of the box way to just transfer that
but without all the other GSSAPI stuff?

Do you think it will be possible to have still only one `ssh`, `scp`,
etc. command and that will just use extra GSSAPI stuff if installed and
needed by a certain connection?


Cheers,
Chris.

Re: Debian openssh option review: considering splitting out GSS-API key exchange

[Damien Miller]

On Tue, 2 Apr 2024, Colin Watson wrote:

[I'm not subscribed to the debian-* lists, please Cc me in replies if
you want me to see them]

> [I've CCed openssh-unix-dev for awareness, but set Mail-Followup-To to
> just debian-devel and debian-ssh to avoid potentially spamming them
> with a long discussion. If you choose to override this then that's
> your call, but please be mindful of upstream's time.]

Thanks Colin for considering how to reduce dependency chains for sshd.
I just remembered that this is not the first time that sshd has been
attacked via a transitive library dependency - it has happened before,
about 10 years ago:

https://www.welivesecurity.com/2014/02/21/an-in-depth-analysis-of-linuxebury/

Attacks like these are impossible for sshd to defend against itself.
Instead we have to look to minimising the number of libraries that end
up in sshd's address space, especially that of the privileged sshd
process.

We are currently exploring splitting sshd into separate binaries for
the listener, privileged monitor, pre- and post-auth network-facing
processes so that each can be reduced in size and functionality to
the minimum possible. This should remove a number of dependencies from
the privileged process. There's a draft of these changes at
https://github.com/djmdjm/openssh-wip/pull/26 but it's OpenBSD-only
at this stage. We're likely to proceed with splitting the listener
process from the rest of sshd hopefully before the next release.

Another thing we're considering in OpenSSH is changing how we integrate
with PAM. PAM's API demands loading modules into the authenticating
process' address space, but obviously we've just been reminded that this
is risky.

I think that I would prefer to move to a model where there PAM auth and
account modules run in a helper process, and only the session module
runs in the unprivileged post-auth sshd process.

This means that PAM auth/account modules and their transitive library
dependencies cannot affect the sshd address space. They would still
likely need to run with privilege, could still fail permissively in
unwanted situations and might still be able to cause problems directly
(e.g. opening a reverse shell from the PAM module itself), but they
would no longer have direct access to the contents of sshd network
traffic, signatures, etc that are extremely useful in building NOBUS
(https://en.wikipedia.org/wiki/NOBUS) backdoors like the xv one.

Where this gets challenging is that some PAM modules make assumptions
that the auth, account and session modules all run in the same address
space. These would break until re-architected to pass things explicitly,
e.g. via environment variables, temp files, etc.

Time permitting, I'll get a prototype of these changes made for wider
experimentation.

-d

Debian openssh option review: considering splitting out GSS-API key exchange

[Colin Watson]

[I've CCed openssh-unix-dev for awareness, but set Mail-Followup-To to
just debian-devel and debian-ssh to avoid potentially spamming them with
a long discussion.  If you choose to override this then that's your
call, but please be mindful of upstream's time.]

Following the xz-utils backdoor, I'm reconsidering some choices in
Debian's OpenSSH packaging.  Please note that significant rearchitecture
of the upstream code is out of scope for the Debian packaging, so I'm
going to disregard comments of the form "maybe there should be a module
loader so all these things can just be plug-ins" or other such blue-sky
things; from my point of view this is just about configuring things a
bit more wisely within more or less our current constraints.


libsystemd
==

This is the obvious thing on everyone's mind right now.  At the time I
merged that patch, "not NIHing code that's in a perfectly good library"
seemed like a reasonable trade-off, but we do seem to have ended up on
the wrong side of history on this one.  There's work in progress to land
readiness protocol notification upstream without libsystemd (thanks
Damien and Luca!), and I expect to cherry-pick this into Debian once
it's agreed, so we'll get rid of that linkage and reduce our patch load
a bit.

We also have a patch from Ubuntu to support the systemd socket
activation protocol.  I've rewritten this to avoid using libsystemd, and
I'll submit it upstream once the readiness notification work is sorted
out.  But it's not particularly invasive once the libsystemd linkage is
removed, so it's not the end of the world if this ends up staying in our
patch queue.


GSS-API key exchange


Way back in 2005, I merged the GSS-API key exchange patch into Debian's
main openssh package (https://bugs.debian.org/275472).  At the time it
seemed like a sensible overall reduction in maintenance burden (if I
remember correctly, the openssh-krb5 package often ended up lagging a
fair bit behind openssh).  While the patch is fairly large, it hasn't
generally been too hard to forward-port to newer versions of OpenSSH,
and Fedora carries it too so there's some sharing of work.

However, OpenSSH upstream has long rejected it, mainly on the basis that
they don't like adding new pre-authentication attack surface, and this
week seems like a good one to reconsider what patches we're shipping by
default.  gssapi.patch is the largest patch in our openssh package by an
order of magnitude, and easily the most intrusive in terms of complexity
and exposure, so I've somewhat regretted my choice to merge it a few
times over the years.

All the same, I'm aware that some people now depend on having this
facility in Debian's main openssh package: I get enough occasional bug
reports to convince me that it's still in use.  So, if I decide to split
it back out, I'd want to arrange for a somewhat graceful transition.
We've had it for nearly 20 years now, so we can take the time to do a
proper job that at least tries not to leave users in the lurch.

How does this rough plan sound?

 * for Debian trixie (current testing):

   * add dependency-only packages called something like
 openssh-client-gsskex and openssh-server-gsskex, depending on their
 non-gsskex alternatives
   * add NEWS.Debian entry saying that people need to install these
 packages if they want to retain GSS-API key exchange support
   * add release note saying the same

 * for Debian trixie+1 (or maybe after the next Ubuntu LTS, depending on
   exact timings):

   * add separate openssh-gsskex source package, carrying gssapi.patch
 in addition to whatever's in openssh, and whose binary packages
 Conflicts/Replaces/Provides the corresponding ones from openssh
   * add some kind of regular CI to warn about openssh-gsskex being out
 of date relative to openssh
   * drop gssapi.patch from openssh, except for small patches to
 configuration file handling to accept the relevant options with
 some kind of informative warning (compare
 https://bugs.debian.org/152657)

I guess we should decide whether the separate packages are to be needed
for GSS-API authentication as well as key exchange, because that affects
the choice of dependency-only package names in trixie.  If we only split
out gssapi.patch (for key exchange; sorry about the slightly misleading
name) but kept --with-kerberos5 (which also controls authentication),
then we'd significantly reduce our patch load but not sshd's linkage.

I've seen the suggestion of using libgssglue here
(https://fosstodon.org/@jas/112194876950058188).  That might be a good
idea and I have no particular objection to it, though I also don't know
much about it and it would probably be better if an expert did the work.
Perhaps it would make continuing to build the default variant using
--with-kerberos5 more palatable, since then the extra non-trivial
linkage would only affect people who turn on those options.


TCP wrappers


We carry a