Bug#453241: still broken (and partly openssh's fault)

[Russ Allbery]

Richard A Nelson <[EMAIL PROTECTED]> writes:

> It got further (into pam_session), and having learned from Russ, I fired
> up valgrind and found it failing in another MIT routine, but after
> rebuilding libpam_heimdal, it looks like things are once again golden :)

Brian, you may want to request binNMUs from the release team for every
package that depends on Heimdal on all platforms other than i386 (I don't
think any of the other platforms in Debian allow shared objects with
non-PIC code, but I could be wrong).  Rebuilds are needed to pick up the
new library versioning.

-- 
Russ Allbery ([EMAIL PROTECTED])   



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: [JIRA] Created: (HEIMDAL-117) Re: [Heimdal-bugs] Bug#453241: still broken (and partly openssh's fault)

[Love Hörnquist Åstrand]

30 apr 2008 kl. 23.39 skrev Russ Allbery:


Love Hörnquist Åstrand <[EMAIL PROTECTED]> writes:


How about using libtool ? Can you test this patch ?

http://www.h5l.org/fisheye/changelog/heimdal/?cs=23146


Yup, this works, although it produces a bunch of spurious output in  
the
middle of the configure run (I think because libtool is trying to be  
a bit

too smart and report what it's doing).


Great, thanks for the confirmation.

Love




--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: [JIRA] Created: (HEIMDAL-117) Re: [Heimdal-bugs] Bug#453241: still broken (and partly openssh's fault)

[Russ Allbery]

Love Hörnquist Åstrand <[EMAIL PROTECTED]> writes:

> How about using libtool ? Can you test this patch ?
>
> http://www.h5l.org/fisheye/changelog/heimdal/?cs=23146

Yup, this works, although it produces a bunch of spurious output in the
middle of the configure run (I think because libtool is trying to be a bit
too smart and report what it's doing).

Thanks!

-- 
Russ Allbery ([EMAIL PROTECTED]) 



--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: [JIRA] Created: (HEIMDAL-117) Re: [Heimdal-bugs] Bug#453241: still broken (and partly openssh's fault)

[Russ Allbery]

Love Hörnquist Åstrand <[EMAIL PROTECTED]> writes:

> How about using libtool ? Can you test this patch ?
>
> http://www.h5l.org/fisheye/changelog/heimdal/?cs=23146

Oh, yeah, that's a good idea.  I'll be able to test this tomorrow evening.

-- 
Russ Allbery ([EMAIL PROTECTED])   



--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: [JIRA] Created: (HEIMDAL-117) Re: [Heimdal-bugs] Bug#453241: still broken (and partly openssh's fault)

[Love Hörnquist Åstrand]

To provide a bit of additional information (I should have pasted in  
the
config.log output as well), the existing check fails on x86_64  
because it
uses -shared to link the test object but doesn't use -fPIC.  This  
works on
x86, but other platforms, such as x86_64, require all objects that  
go into

a shared object be built PIC and the link dies with relocation errors.

There's probably something better to do here than just blindly add - 
fPIC,
based on what the correct PIC flag is for the compiler used, but I  
wasn't

sure how to get at that.


How about using libtool ? Can you test this patch ?

http://www.h5l.org/fisheye/changelog/heimdal/?cs=23146

Love

Bug#453241: still broken (and partly openssh's fault)

[Russ Allbery]

Russ Allbery <[EMAIL PROTECTED]> writes:
> Brian May <[EMAIL PROTECTED]> writes:

>> Can I please confirm what version of Heimdal you are using? The initial
>> bug report seemed to quote the old version in testing, but here you
>> seem to indicate the latest version in unstable. I just want to make
>> sure.
>>
>> As far as I can tell, all exported symbols from libkrb5.24.0.0 use
>> HEIMDAL_KRB5_1.0 for the versioned symbol name.

> I'll check again tonight on amd64.  The problem is specifically on
> amd64; if you're checking on i386, you may not see it.  I wasn't seeing
> any symbol versioning in readelf.

Now I can't get valgrind to give me a backtrace.  :/  It looks like it's
losing its file descriptor just before it would, and the report is
truncated.  I'm not sure what would have changed to cause that; I remember
having to fiddle with it to get it to work the first time, but now I can't
reproduce the environment.

I can confirm that there's no symbol versioning, though; objdump -T shows
no versions on any of the symbols.  And I know that causes problems, so
I'm pretty sure that's the issue still.

Checking the amd64 build logs, I see:

checking for ld --version-script... no

which is doubtless the problem.  Here's a patch that looks like it should
work.  After building Heimdal with this patch, I get symbol versions on
amd64.

--- heimdal-1.1/cf/version-script.m4.orig   2008-01-24 05:13:43.0 
-0800
+++ heimdal-1.1/cf/version-script.m42008-04-28 20:28:39.0 -0700
@@ -16,7 +16,7 @@
 int gss_init_creds(int foo) { return 0; }
 EOF
 
-  if AC_TRY_COMMAND([${CC-cc} $CFLAGS $LDFLAGS -shared
+  if AC_TRY_COMMAND([${CC-cc} $CFLAGS $LDFLAGS -fPIC -shared
-o conftest.so conftest.c
-Wl,--version-script,conftest.map]);
   then
@@ -37,4 +37,4 @@
 AM_CONDITIONAL(versionscript,test $doversioning = yes)
 AC_SUBST(LDFLAGS_VERSION_SCRIPT)
 
-])
\ No newline at end of file
+])

-- 
Russ Allbery ([EMAIL PROTECTED])   

Bug#453241: still broken (and partly openssh's fault)

[Brian May]

Richard A Nelson wrote:
> Indeed, I have no issues on i368, only amd64... though even on i386,
> I see a few @HEIMDAL_X509_1.0,
>
> $readelf -s /usr/lib/libkrb5.so.24  | grep HEIMDAL_
> $readelf -s /usr/lib/libheimntlm.so.0 | grep HEIMDAL_
> $readelf -s /usr/lib/libhx509.so.3 | grepp HEIMDAL_
This was on amd64, right?

In which case maybe the build system in Heimdal is broken somehow on
AMD64 and not including the versioned symbols correctly.

Brian May



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: still broken (and partly openssh's fault)

[Richard A Nelson]

On Mon, 28 Apr 2008, Russ Allbery wrote:


Brian May <[EMAIL PROTECTED]> writes:


Can I please confirm what version of Heimdal you are using? The initial
bug report seemed to quote the old version in testing, but here you seem
to indicate the latest version in unstable. I just want to make sure.

As far as I can tell, all exported symbols from libkrb5.24.0.0 use
HEIMDAL_KRB5_1.0 for the versioned symbol name.


ii  heimdal-client 1.1-2


I'll check again tonight on amd64.  The problem is specifically on amd64;
if you're checking on i386, you may not see it.  I wasn't seeing any
symbol versioning in readelf.


Indeed, I have no issues on i368, only amd64... though even on i386,
I see a few @HEIMDAL_X509_1.0,

$readelf -s /usr/lib/libkrb5.so.24  | grep HEIMDAL_
$readelf -s /usr/lib/libheimntlm.so.0 | grep HEIMDAL_
$readelf -s /usr/lib/libhx509.so.3 | grepp HEIMDAL_

Whereas on i386, I see HEIMDAL_KRB5_1.0, and HEIMDAL_X509_1.0


If OpenSSH is linked against MIT Kerberos, like you say, then simply
proving that the segfault occurs inside MIT Kerberos is insufficient,
unfortunately, because we have to expect OpenSSH may call MIT Kerberos
functions at some point.


In which case, the issue should show up on i386 as well, no?


According to valgrind, the backtrace showed the segfaults definitely in
functions called by libpam-heimdal, not by openssh itself.  I'll include
the backtrace when I get home and can reproduce it.

gdb doesn't produce a usable backtrace (probably because of the library
confusion).  Only valgrind would work for me, and only with a rebuilt
libpam-heimdal with debugging information.


yes, I recompiled libpam-heimdal and ssh with debugging - gdb gave no
helpful information at all, and I never thought to try valgrind

--
Rick Nelson
Life'll kill ya -- Warren Zevon
Then you'll be dead -- Life'll kill ya



--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: still broken (and partly openssh's fault)

[Russ Allbery]

Brian May <[EMAIL PROTECTED]> writes:

> Can I please confirm what version of Heimdal you are using? The initial
> bug report seemed to quote the old version in testing, but here you seem
> to indicate the latest version in unstable. I just want to make sure.
>
> As far as I can tell, all exported symbols from libkrb5.24.0.0 use
> HEIMDAL_KRB5_1.0 for the versioned symbol name.

I'll check again tonight on amd64.  The problem is specifically on amd64;
if you're checking on i386, you may not see it.  I wasn't seeing any
symbol versioning in readelf.

> If OpenSSH is linked against MIT Kerberos, like you say, then simply
> proving that the segfault occurs inside MIT Kerberos is insufficient,
> unfortunately, because we have to expect OpenSSH may call MIT Kerberos
> functions at some point.

According to valgrind, the backtrace showed the segfaults definitely in
functions called by libpam-heimdal, not by openssh itself.  I'll include
the backtrace when I get home and can reproduce it.

gdb doesn't produce a usable backtrace (probably because of the library
confusion).  Only valgrind would work for me, and only with a rebuilt
libpam-heimdal with debugging information.

-- 
Russ Allbery ([EMAIL PROTECTED])   



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: still broken (and partly openssh's fault)

[Brian May]

Brian May wrote:
> It occurred to me that the stack trace is probably in the Debian bug
> report, I will check that now.
>   
I can't seem to find the stack trace with debugging information, if you
still have a copy can you please send it to the BTS?

Thanks.



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: still broken (and partly openssh's fault)

[Brian May]

Russ Allbery wrote:
> I spent an hour this evening tracking this down.  The problem is that
> Heimdal isn't using symbol versioning in its shared libraries.
> libpam-heimdal therefore binds to unversioned symbols, which works fine if
> the calling program doesn't load any other Kerberos library.  However,
> OpenSSH is linked with MIT Kerberos, and therefore at run time the
> unversioned libpam-heimdal symbols are bound to the MIT Kerberos version
> of libkrb5 which is already loaded in memory and chaos ensues.  valgrind
> was the debugging tool that finally gave me the necessary clue.  The
> segfault kept showing up with backtraces inside libkrb5.3.3 instead of
> libkrb5.24.0.0.
>   
Can I please confirm what version of Heimdal you are using? The initial
bug report seemed to quote the old version in testing, but here you seem
to indicate the latest version in unstable. I just want to make sure.

As far as I can tell, all exported symbols from libkrb5.24.0.0 use
HEIMDAL_KRB5_1.0 for the versioned symbol name.

objdump -T libkrb5.so.24.0.0

...

0001d180 gDF .text  003e  HEIMDAL_KRB5_1.0 
krb5_config_vget_string_default

00047140 gDF .text  0034  HEIMDAL_KRB5_1.0 krb5_rd_req_out_get_ticket

00028130 gDF .text  0074  HEIMDAL_KRB5_1.0 krb5_digest_free

0004cc30 gDF .text  00c6  HEIMDAL_KRB5_1.0 krb5_storage_emem

0004b100 gDF .text  0037  HEIMDAL_KRB5_1.0 _krb5_get_int

0002c5b0 gDF .text  022e  HEIMDAL_KRB5_1.0 
krb5_get_credentials_with_flags

0001ca10 gDF .text  0038  HEIMDAL_KRB5_1.0 krb5_encode_EncTGSRepPart

0001eba0 gDF .text  007e  HEIMDAL_KRB5_1.0 
krb5_prepend_config_files_default

0001a530 gDF .text  00f5  HEIMDAL_KRB5_1.0 krb5_cc_retrieve_cred

0001cfe0 gDF .text  0034  HEIMDAL_KRB5_1.0 krb5_config_get_time

0002dfa0 gDF .text  0208  HEIMDAL_KRB5_1.0 _krb5_get_host_realm_int

0004b9c0 gDF .text  009f  HEIMDAL_KRB5_1.0 krb5_ret_times

000165c0 gDF .text  0033  HEIMDAL_KRB5_1.0 krb5_sockaddr_uninteresting

0002fd00 gDF .text  0080  HEIMDAL_KRB5_1.0 krb5_get_in_tkt_with_keytab

00017d30 gDF .text  0039  HEIMDAL_KRB5_1.0 krb5_address_compare

0003d950 gDF .text  0038  HEIMDAL_KRB5_1.0 krb5_c_enctype_compare

0001e7d0 gDF .text  0139  HEIMDAL_KRB5_1.0 
krb5_get_default_in_tkt_etypes

000450d0 gDF .text  0021  HEIMDAL_KRB5_1.0 krb5_unparse_name_fixed_short


If OpenSSH is linked against MIT Kerberos, like you say, then simply
proving that the segfault occurs inside MIT Kerberos is insufficient,
unfortunately, because we have to expect OpenSSH may call MIT Kerberos
functions at some point.
> This is a bug in the Debian Heimdal packages, I believe.  They used to use
> symbol versioning precisely because of this problem; see Bug#205592 which
> was closed in 0.6-4.  It looks like that was lost or dropped somewhere
> along the way.
>   
The symbol versioning was moved to the upstream code; I don't guarantee
that they got it right, but I want some evidence before I forward this
upstream.

It occurred to me that the stack trace is probably in the Debian bug
report, I will check that now.

Brian May



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: still broken (and partly openssh's fault)

[Richard A Nelson]

On Mon, 21 Apr 2008, Russ Allbery wrote:


I spent an hour this evening tracking this down.  The problem is that
Heimdal isn't using symbol versioning in its shared libraries.
libpam-heimdal therefore binds to unversioned symbols, which works fine if
the calling program doesn't load any other Kerberos library.  However,
OpenSSH is linked with MIT Kerberos, and therefore at run time the
unversioned libpam-heimdal symbols are bound to the MIT Kerberos version
of libkrb5 which is already loaded in memory and chaos ensues.  valgrind
was the debugging tool that finally gave me the necessary clue.  The
segfault kept showing up with backtraces inside libkrb5.3.3 instead of
libkrb5.24.0.0.


Thanks for the effort - and the education,  I've used valgrind, but
never for something like this


This is a bug in the Debian Heimdal packages, I believe.  They used to use
symbol versioning precisely because of this problem; see Bug#205592 which
was closed in 0.6-4.  It looks like that was lost or dropped somewhere
along the way.


Most likely with the recent bump to the 1.x series - looks like a big
source and packaging change; I ran into another fallout of the packaging
change (already fixed)


I'm copying Brian May on this.  I think the bug should probably be
reassigned to the heimdal source package.


Reassigned...   Fortunately, there aren't that many packages that depend
upon Heimdal, as they'll all need rebuilding after Heimdal is updated.

--
Rick Nelson
Life'll kill ya -- Warren Zevon
Then you'll be dead -- Life'll kill ya



--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: still broken (and partly openssh's fault)

[Russ Allbery]

Richard A Nelson <[EMAIL PROTECTED]> writes:

> I found some odd news!
>
> After building openssh on a recent Sid box, the pam account module
> now works as expected - no segv :)  This, along with the fact that
> there's a new openssh version (with supposed changes in this area),
> makes it seem like getting ssh rebuilt soon is a good idea !
>
> However, the session module still blows chunks, and since it is called
> by the subordinate (unpriviledged) thread, I don't know how to trap it:

I spent an hour this evening tracking this down.  The problem is that
Heimdal isn't using symbol versioning in its shared libraries.
libpam-heimdal therefore binds to unversioned symbols, which works fine if
the calling program doesn't load any other Kerberos library.  However,
OpenSSH is linked with MIT Kerberos, and therefore at run time the
unversioned libpam-heimdal symbols are bound to the MIT Kerberos version
of libkrb5 which is already loaded in memory and chaos ensues.  valgrind
was the debugging tool that finally gave me the necessary clue.  The
segfault kept showing up with backtraces inside libkrb5.3.3 instead of
libkrb5.24.0.0.

The specific crashes that you're seeing are inside the profile library
calls, but that's just because that's the first significant Kerberos
library code that the PAM module calls that differs between MIT Kerberos
and Heimdal.

This is a bug in the Debian Heimdal packages, I believe.  They used to use
symbol versioning precisely because of this problem; see Bug#205592 which
was closed in 0.6-4.  It looks like that was lost or dropped somewhere
along the way.

I'm copying Brian May on this.  I think the bug should probably be
reassigned to the heimdal source package.

-- 
Russ Allbery ([EMAIL PROTECTED])   



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Bug#453241: still broken (and partly openssh's fault)

[Richard A Nelson]

On Sun, 13 Apr 2008, Matthijs Mohlmann wrote:


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

Can you try with ssh and debugging symbols on ? Probably you get more
information with gdb then.


Being that there are no debug packages for openssh, I rebuilt it
on a current Sid machine (with debugging enabled).


I'll ask also the upstream maintainer.

Russ: Can you take a look at this bug report. It seems that sshd is
segfaulting on AMD64 when using the libpam-heimdal module. I do not have
a AMD64 box, so it's almost impossible for me to test.


I found some odd news!

After building openssh on a recent Sid box, the pam account module
now works as expected - no segv :)  This, along with the fact that
there's a new openssh version (with supposed changes in this area),
makes it seem like getting ssh rebuilt soon is a good idea !

However, the session module still blows chunks, and since it is called
by the subordinate (unpriviledged) thread, I don't know how to trap it:

Accepted publickey for renegade from 9.30.102.134 port 53147 ssh2
debug1: monitor_child_preauth: renegade has been authenticated by
privileged process
debug3: mm_get_keystate: Waiting for new keys
debug3: mm_request_receive_expect entering: type 25
debug3: mm_request_receive entering
debug3: mm_newkeys_from_blob: 0x7fee6df93ed0(128)
debug2: mac_setup: found hmac-md5
debug3: mm_get_keystate: Waiting for second key
debug3: mm_newkeys_from_blob: 0x7fee6df93ed0(128)
debug2: mac_setup: found hmac-md5
debug3: mm_get_keystate: Getting compression state
debug3: mm_get_keystate: Getting Network I/O buffers
debug3: mm_share_sync: Share sync
debug3: mm_share_sync: Share sync end
debug1: temporarily_use_uid: 2007/2000 (e=0/2000)
debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
debug1: restore_uid: 0/2000
debug3: PAM: opening session
debug2: User child is on pid 30175
debug3: mm_request_receive entering
debug1: do_cleanup
debug1: PAM: cleanup
debug3: PAM: sshpam_thread_cleanup entering

Program exited with code 0377.

Note that it also fails if I do use GSSAPI (instead of ssh key, like
the example shown above).



Regards,

Matthijs Mohlmann

Richard Nelson wrote:

Ah, a little more information - this segv only happens when using
password authentication (ssh keys work fine)

sshd_config has
UsePAM yes
PubkeyAuthentication yes
PasswordAuthentication yes
ChallengeResponseAuthentication no

Richard Nelson wrote:

# /usr/sbin/sshd -Dddd >~/log 2>&1
Segmentation fault

The last lines of log:
debug3: mm_auth_password entering
debug3: mm_request_send entering: type 11
debug3: mm_auth_password: waiting for MONITOR_ANS_AUTHPASSWORD
debug3: mm_request_receive_expect entering: type 12
debug3: mm_request_receive entering
debug3: monitor_read: checking request 11
debug1: do_cleanup
debug1: PAM: cleanup
debug3: PAM: sshpam_thread_cleanup entering

gdb isn't very helpful
Program received signal SIGSEGV, Segmentation fault.
0x2acda6fe7af2 in ?? ()
(gdb) bt
#0  0x2acda6fe7af2 in ?? ()
#1  0x2acda692ad86 in ?? ()
#2  0x0050 in ?? ()
#3  0x0001 in ?? ()
#4  0x7fff05c7cf10 in ?? ()
#5  0x in ?? ()
(gdb) quit
The program is running.  Exit anyway? (y or n) y
debug1: do_cleanup
debug1: PAM: cleanup
debug3: PAM: sshpam_thread_cleanup entering

I installed libpam-dbg, but still didn't get any information

removing pam_krb5 from /etc/pam.d/common-auth fixes the problem







-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIAb1n2n1ROIkXqbARAuG7AJ9glEncS6jvQie2UhnY4ya5Tk91HACbBKEp
sgyobGhwwaO6vxCDg4TQb0U=
=9KMZ
-END PGP SIGNATURE-



--
Rick Nelson
Life'll kill ya -- Warren Zevon
Then you'll be dead -- Life'll kill ya



--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]