kpasswdd dumps on OpenBSD6.3

[ASV]

Hi everyone,
I'm experiencing a frustrating issue with Heimdal 7.5.0p0 installed on
OpenBSD 6.3 and at this point I'm at a dead end.
Everything is working well but kpasswdd. Everytime I try to reset a
password from any computer after few seconds the daemon core dumps!

(ON A WORKSTATION)
% kpasswd
a...@bla.net's Password: 
New password: 
Verify password - New password:
(IT GETS STUCK FOR A FEW SECONDS HERE AND THEN ...)
kpasswd: krb5_set_password_using_ccache: Unable to reach any changepw
server  in realm BLA.NET

DNS is OK and the server is powerful enough (16GB of storage, 1GB RAM
[+600mb still free when this happens]).
I've tried to deinstall the standard package and libraries and re-
install from ports (recompiled it all) but nothing changes.
I've tried to get a little info more:

(ON THE SERVER)
# gdb --args /usr/local/heimdal/libexec/kpasswdd -r BLA.NET 
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are welcome to change it and/or distribute copies of it under
certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "amd64-unknown-openbsd6.3"...(no debugging
symbols found)

(gdb) r
Starting program: /usr/local/heimdal/libexec/kpasswdd -r BLA.NET

(THE FOLLOWING APPEARS AT THE PASSWORD CHANGE ATTEMPT)
 
Program received signal SIGSEGV, Segmentation fault.
0x1db26a5615d6 in change () from
/usr/local/heimdal/lib/libkadm5srv.so.3.0
Current language:  auto; currently minimal
(gdb) The program is running.  Exit anyway? (y or n) n
Not confirmed.
(gdb) backtrace
#0  0x1db26a5615d6 in change () from
/usr/local/heimdal/lib/libkadm5srv.so.3.0
#1  0x1db26a56155f in kadm5_s_chpass_principal_cond () from
/usr/local/heimdal/lib/libkadm5srv.so.3.0
#2  0x1dafc3001b90 in ?? () from
/usr/local/heimdal/libexec/kpasswdd
#3  0x1dafc3000996 in ?? () from
/usr/local/heimdal/libexec/kpasswdd
#4  0x in ?? ()

Any help would be highly appreciated.
Thanks in advance.

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

Wno-unused-parameter -Wmissing-prototypes -Wpointer-arith 
-Wbad-function-cast -Wmissing-declarations -Wnested-externs -Wshadow -Wno-extra 
-Wno-missing-field-initializers -Wno-strict-aliasing -Wno-shadow -DINET6 -g 
-ggdb3 -O0 nt_gss_client.o nt_gss_common.o common.o 
/usr/ports/pobj/heimdal-7.5.0/heimdal-7.5.0/lib/vers/.libs/libvers.a -L.libs 
-lgssapi -lheimntlm -lkrb5 -lasn1 -lcom_err -lroken -lutil -lwind -lheimbase 
-lhx509 -lhcrypto -lcrypto -lheimsqlite -Wl,-rpath,/usr/local/heimdal/lib


On Thu, 2018-08-02 at 12:44 -0400, Viktor Dukhovni wrote:
> > On Aug 2, 2018, at 12:33 PM, ASV  wrote:
> > 
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x1db26a5615d6 in change () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > Current language:  auto; currently minimal
> > (gdb) The program is running.  Exit anyway? (y or n) n
> > Not confirmed.
> > (gdb) backtrace
> > #0  0x1db26a5615d6 in change () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > #1  0x1db26a56155f in kadm5_s_chpass_principal_cond () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > #2  0x1dafc3001b90 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #3  0x1dafc3000996 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #4  0x in ?? ()
> > 
> > Any help would be highly appreciated.
> > Thanks in advance.
> 
> 1.  Please rebuild heimdal with debugging symbols:
> 
>   CFLAGS="-g -ggdb3 -O0"
> 
> that should make possible a stack trace with line numbers, ...
> 
> 2.  What happens when you try to change the password as root via
> "kadmin -l cpw ..."?  Does that work?
> 
> 3.  It might be helpful to know what HDB backend you're using.
> 
> 4.  Even without symbols you can ask GDB for the faulting
> instruction:
> 
>   x/i $pc
> 
> and print the register values:
> 
>   i reg
> 
> This may help to understand whether this a NULL pointer dereference,
> or something else.
> 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

5073165aed820)
 41873 kpasswdd RET   kbind 0
 41873 kpasswdd CALL  kbind(0x7f7f8290,24,0xc5f5073165aed820)
 41873 kpasswdd RET   kbind 0
 41873 kpasswdd CALL  kbind(0x7f7f8290,24,0xc5f5073165aed820)
 41873 kpasswdd RET   kbind 0
 41873 kpasswdd PSIG  SIGSEGV SIG_DFL code SEGV_MAPERR<1> addr=0xc8 trapno=6
 41873 kpasswdd NAMI  "kpasswdd.core"



On Fri, 2018-08-03 at 10:59 -0400, Viktor Dukhovni wrote:
> > On Aug 3, 2018, at 5:27 AM, ASV  wrote:
> > 
> > Changing password as suggested (kadmin -l cpw ...) works
> 
> No crashes, right?  Modulo authentication of the user, UDP transport,
> ...
> ultimately "kadmin -l cpw" and "kpasswd" should end up calling the
> same change() function in much the same way.
> 
> > After starting back the daemon it worked well.
> 
> What does "it worked well" mean?
> 
> > I'm using the HDB3 which I believe is part of the package itself
> > (if I properly understood your question).
> 
> Yes, thanks.
> 
> > This is the output of the procedure on the newly re-compiled
> > heimdal with the CFLAGS="-g -ggdb3 -O0" but doesn't look much
> > different from the previous non-debugging version to me (hopefully
> > I didn't make any mistake).
> 
> It looks like something stripped the built binaries, please make sure
> your
> build process does not strip the binaries, and that you're running
> the
> binaries with the debug symbols.
> 
> > A snippet of the compilation at the very end of this email.
> > 
> > (gdb) bt
> > #0  0x0ae0709ed5d6 in change () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > #1  0x0ae0709ed55f in kadm5_s_chpass_principal_cond () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0  
> >   
> > #2  0x0ade6d002bfb in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #3  0x0ade6d001b89 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #4  0x0ade6d0017d9 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #5  0x0ade6d001073 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #6  0x0ade6d0009a6 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #7  0x in ?? ()
> > (gdb) x/i $pc
> > 0xae0709ed5d6 : cmpl   $0x0,0xc8(%r14)
> 
> This is a structure member dereference at offset 200 from
> a structure at $r14, which we see below is a NULL pointer:
> 
> > (gdb) i reg
> > rax0x76118487c2646c07   8507726889696390151
> > rbx0x1  1
> > rcx0x0  0
> > rdx0x1  1
> > rsi0xae13fbbe1a011962553196960
> > rdi0x0  0
> > rbp0x7f7d58b0   0x7f7d58b0
> > rsp0x7f7d57a0   0x7f7d57a0
> > r8 0x0  0
> > r9 0xae13fbbe4e011962553197792
> > r100xae0a1066ab011959890504368
> > r110xae0a299e00011959916945408
> > r120xae13fbbe1a011962553196960
> > r130x0  0
> > r140x0  0
> > r150xae13fbbe4e011962553197792
> > rip0xae0709ed5d60xae0709ed5d6 
> 
> Looking at the code for change(), this seems to be the
> test condition for block:
> 
> if (!context->keep_open) {
> ret = context->db->hdb_open(context->context, context->db,
> O_RDWR, 0);
> if(ret)
> return ret;
> }
> 
> So somehow the server_handle passed to change() is NULL.  Which
> means that the kadm5_handle that kpasswdd passes to
> kadm5_s_chpass_principal_cond() is NULL.  The handle is initialized
> via:
> 
> ret = kadm5_init_with_password_ctx(context,
>admin,
>NULL,
>KADM5_ADMIN_SERVICE,
>, 0, 0,
>_handle);
> 
> And failure short-circuits the call to
> kadm5_s_chpass_principal_cond()
> so it is not clear how this could happen.  Debugging symbols are
> needed.
> 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

I think that I finally got it, did I?


Program received signal SIGSEGV, Segmentation fault.
0x07fe9c69d5d6 in change () from /usr/local/heimdal/lib/libkadm5srv.so.3.0
Current language:  auto; currently minimal
(gdb) bt
#0  0x07fe9c69d5d6 in change () from 
/usr/local/heimdal/lib/libkadm5srv.so.3.0
#1  0x07fe9c69d55f in kadm5_s_chpass_principal_cond () from 
/usr/local/heimdal/lib/libkadm5srv.so.3.0
#2  0x07fbfc702bfb in ?? () from /usr/local/heimdal/libexec/kpasswdd
#3  0x07fbfc701b89 in ?? () from /usr/local/heimdal/libexec/kpasswdd
#4  0x07fbfc7017d9 in ?? () from /usr/local/heimdal/libexec/kpasswdd
#5  0x07fbfc701073 in ?? () from /usr/local/heimdal/libexec/kpasswdd
#6  0x07fbfc7009a6 in ?? () from /usr/local/heimdal/libexec/kpasswdd
#7  0x in ?? ()
(gdb) x/i $pc   


0x7fe9c69d5d6 : cmpl   $0x0,0xc8(%r14)
(gdb) x/40i $pc-102
0x7fe9c69d570 : push   %rbp
0x7fe9c69d571 :   mov%rsp,%rbp
0x7fe9c69d574 :   push   %r15
0x7fe9c69d576 :   push   %r14
0x7fe9c69d578 :   push   %r13
0x7fe9c69d57a :  push   %r12
0x7fe9c69d57c :  push   %rbx
0x7fe9c69d57d :  sub$0xe8,%rsp
0x7fe9c69d584 :  mov%r9,%r15
0x7fe9c69d587 :  mov%r8,%r13
0x7fe9c69d58a :  mov%ecx,0xff04(%rbp)
0x7fe9c69d590 :  mov%edx,%ebx
0x7fe9c69d592 :  mov%rsi,%r12
0x7fe9c69d595 :  mov%rdi,%r14
0x7fe9c69d598 :  mov2174785(%rip),%rax# 
0x7fe9c8b04e0 <__guard_local>
0x7fe9c69d59f :  mov%rax,0xffd0(%rbp)
0x7fe9c69d5a3 :  xorps  %xmm0,%xmm0
0x7fe9c69d5a6 :  movaps %xmm0,0xffc0(%rbp)
0x7fe9c69d5aa :  movaps %xmm0,0xffb0(%rbp)
0x7fe9c69d5ae :  movaps %xmm0,0xffa0(%rbp)
0x7fe9c69d5b2 :  movaps %xmm0,0xff90(%rbp)
0x7fe9c69d5b6 :  movaps %xmm0,0xff80(%rbp)
0x7fe9c69d5ba :  movaps %xmm0,0xff70(%rbp)
0x7fe9c69d5c1 :  movaps %xmm0,0xff60(%rbp)
0x7fe9c69d5c8 :  movaps %xmm0,0xff50(%rbp)
0x7fe9c69d5cf :  movaps %xmm0,0xff40(%rbp)
0x7fe9c69d5d6 : cmpl   $0x0,0xc8(%r14)
0x7fe9c69d5de : je 0x7fe9c69d60f 
0x7fe9c69d5e0 : mov%r13,0xfef0(%rbp)
0x7fe9c69d5e7 : mov%r14,%rdi
0x7fe9c69d5ea : callq  0x7fe9c69cf00 <_init+3104>
0x7fe9c69d5ef : mov%eax,%r13d
0x7fe9c69d5f2 : test   %r13d,%r13d
0x7fe9c69d5f5 : je 0x7fe9c69d64a 
0x7fe9c69d5f7 : cmpl   $0x0,0xc8(%r14)
0x7fe9c69d5ff : je 0x7fe9c69d88f 
0x7fe9c69d605 : mov%r13d,%edi
0x7fe9c69d608 : callq  0x7fe9c69c670 <_init+912>
0x7fe9c69d60d : jmp0x7fe9c69d627 
0x7fe9c69d60f : mov(%r14),%rdi

On Sat, 2018-08-04 at 14:45 -0400, Viktor Dukhovni wrote:
> > On Aug 4, 2018, at 2:36 PM, ASV  wrote:
> > 
> > You've been 23 secs faster to reply than me to correct the email
> > with
> > the wrong output! :D
> > 
> > So, the correct one has been sent already. About the source code,
> > well
> > I've tried with both but the upstream didn't compile so I'll stick
> > to
> > the OpenBSD one from ports.
> 
> Yes, that's best.  Build that with debugging symbols, and make sure
> that the build does not then strip the binaries.
> 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

On Fri, 2018-08-03 at 14:05 -0400, Viktor Dukhovni wrote:
> > On Aug 3, 2018, at 1:43 PM, ASV  wrote:
> > 
> > No, no crashes using "kadmin -l".
> 
> This shows that "kadmin" and the libkadm5srv.so library work fine.
> 
> Which libraries is "kadmin" linked with (post ldd output)?
# ldd /usr/local/heimdal/bin/kadmin
/usr/local/heimdal/bin/kadmin:
StartEnd  Type  Open Ref GrpRef Name
0686bb60 0686bb81c000 exe   20   0  
/usr/local/heimdal/bin/kadmin
068994916000 068994b23000 rlib  01   0  
/usr/local/heimdal/lib/libkadm5clnt.so.3.0
06891523 068915434000 rlib  09   0  
/usr/local/lib/libcom_err.so.21.0
0688da241000 0688da4d7000 rlib  04   0  
/usr/local/heimdal/lib/libkrb5.so.22.0
068944098000 068944342000 rlib  07   0  
/usr/local/heimdal/lib/libasn1.so.22.0
0688fd6d7000 0688fd8ec000 rlib  09   0  
/usr/local/heimdal/lib/libroken.so.3.0
0689b9623000 0689b983 rlib  010   0  
/usr/lib/libutil.so.13.0
0688fe849000 0688fea73000 rlib  06   0  
/usr/local/heimdal/lib/libwind.so.3.0
0689b15ca000 0689b17da000 rlib  07   0  
/usr/local/heimdal/lib/libheimbase.so.2.0
06896c15e000 06896c3b2000 rlib  05   0  
/usr/local/heimdal/lib/libhx509.so.1.0
0688f28ff000 0688f2b41000 rlib  06   0  
/usr/local/heimdal/lib/libhcrypto.so.0.0
068975d14000 0689760f rlib  07   0  
/usr/lib/libcrypto.so.43.1
06892568c000 06892598a000 rlib  05   0  
/usr/local/heimdal/lib/libheimsqlite.so.1.0
0689af3af000 0689af5c5000 rlib  01   0  
/usr/local/heimdal/lib/libkadm5srv.so.3.0
06899ac6a000 06899ae91000 rlib  02   0  
/usr/local/heimdal/lib/libhdb.so.3.0
0689b368f000 0689b3895000 rlib  01   0  
/usr/local/heimdal/lib/libsl.so.1.0
0688ca70a000 0688ca941000 rlib  02   0  
/usr/local/heimdal/lib/libheimedit.so.1.0
06890dd52000 06890dfa8000 rlib  03   0  
/usr/lib/libcurses.so.14.0
068923cc5000 068923ece000 rlib  013   0  
/usr/lib/libpthread.so.25.1
0688eb50d000 0688eb7ed000 rlib  01   0  
/usr/lib/libc.so.92.3
068924a0 068924a0 ld.so 01   0  
/usr/libexec/ld.so
> Which libraries is "kpasswdd" linked with (post ldd output) and make
> sure you're reporting the results for the right executable.
# ldd /usr/local/heimdal/bin/kpasswd
/usr/local/heimdal/bin/kpasswd:
StartEnd  Type  Open Ref GrpRef Name
19fca7d0 19fca7f04000 exe   20   0  
/usr/local/heimdal/bin/kpasswd
19ff4295c000 19ff42bf2000 rlib  01   0  
/usr/local/heimdal/lib/libkrb5.so.22.0
19ff9d5f6000 19ff9d8a rlib  04   0  
/usr/local/heimdal/lib/libasn1.so.22.0
19ff68568000 19ff6876c000 rlib  06   0  
/usr/local/lib/libcom_err.so.21.0
19fed6dab000 19fed6fc rlib  06   0  
/usr/local/heimdal/lib/libroken.so.3.0
19fec573d000 19fec594a000 rlib  07   0  
/usr/lib/libutil.so.13.0
19ff28c6a000 19ff28e94000 rlib  03   0  
/usr/local/heimdal/lib/libwind.so.3.0
19ff6bed7000 19ff6c0e7000 rlib  04   0  
/usr/local/heimdal/lib/libheimbase.so.2.0
19fef522c000 19fef548 rlib  02   0  
/usr/local/heimdal/lib/libhx509.so.1.0
19ffa531e000 19ffa556 rlib  03   0  
/usr/local/heimdal/lib/libhcrypto.so.0.0
19ff51dc7000 19ff521a3000 rlib  04   0  
/usr/lib/libcrypto.so.43.1
19fef6cf8000 19fef6ff6000 rlib  02   0  
/usr/local/heimdal/lib/libheimsqlite.so.1.0
19ff49167000 19ff4937 rlib  09   0  
/usr/lib/libpthread.so.25.1
19ff38f04000 19ff391e4000 rlib  01   0  
/usr/lib/libc.so.92.3
19ff0f00 19ff0f00 ld.so 01   0  
/usr/libexec/ld.so

And yes, these are the right executable called by the rc scripts.

> > With "it worked well" I was just referring to the fact that "kadmin
> > -l cpw ..." doesn't work when kpasswdd is down
> 
> That's not possible.  The "kadmin -l cpw" command makes local changes
> directly in the HDB and does not use "kpasswdd".  Its behaviour
> cannot
> depend on whether kpasswdd is running or not.  It is operates
> silently.
Sorry that was my bad. I've verified it and the only circumstance this
could possibly happen is when the two passwords do not match. So they
did not matc

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

Please DISCARD the previous email where I've erroneously posted the
libraries linked to kpasswd and not to kpasswdd.

On Fri, 2018-08-03 at 14:05 -0400, Viktor Dukhovni wrote:
> > On Aug 3, 2018, at 1:43 PM, ASV  wrote:
> > 
> > No, no crashes using "kadmin -l".
> 
> This shows that "kadmin" and the libkadm5srv.so library work fine.
> 
> Which libraries is "kadmin" linked with (post ldd output)?
# ldd /usr/local/heimdal/bin/kadmin   
/usr/local/heimdal/bin/kadmin:
StartEnd  Type  Open Ref GrpRef Name
031952d0 031952f1c000 exe   20   0  
/usr/local/heimdal/bin/kadmin
031be7982000 031be7b8f000 rlib  01   0  
/usr/local/heimdal/lib/libkadm5clnt.so.3.0
031c15b68000 031c15d6c000 rlib  09   0  
/usr/local/lib/libcom_err.so.21.0
031c1b05f000 031c1b2f5000 rlib  04   0  
/usr/local/heimdal/lib/libkrb5.so.22.0
031bfbe9d000 031bfc147000 rlib  07   0  
/usr/local/heimdal/lib/libasn1.so.22.0
031b81fe8000 031b821fd000 rlib  09   0  
/usr/local/heimdal/lib/libroken.so.3.0
031b99afb000 031b99d08000 rlib  010   0  
/usr/lib/libutil.so.13.0
031c2c46a000 031c2c694000 rlib  06   0  
/usr/local/heimdal/lib/libwind.so.3.0
031c3a5b1000 031c3a7c1000 rlib  07   0  
/usr/local/heimdal/lib/libheimbase.so.2.0
031b96fec000 031b9724 rlib  05   0  
/usr/local/heimdal/lib/libhx509.so.1.0
031ba3106000 031ba3348000 rlib  06   0  
/usr/local/heimdal/lib/libhcrypto.so.0.0
031bb87b9000 031bb8b95000 rlib  07   0  
/usr/lib/libcrypto.so.43.1
031bcda56000 031bcdd54000 rlib  05   0  
/usr/local/heimdal/lib/libheimsqlite.so.1.0
031c30a61000 031c30c77000 rlib  01   0  
/usr/local/heimdal/lib/libkadm5srv.so.3.0
031c213fe000 031c21625000 rlib  02   0  
/usr/local/heimdal/lib/libhdb.so.3.0
031bed128000 031bed32e000 rlib  01   0  
/usr/local/heimdal/lib/libsl.so.1.0
031bef6db000 031bef912000 rlib  02   0  
/usr/local/heimdal/lib/libheimedit.so.1.0
031c26ee6000 031c2713c000 rlib  03   0  
/usr/lib/libcurses.so.14.0
031bbe10 031bbe309000 rlib  013   0  
/usr/lib/libpthread.so.25.1
031b86a1f000 031b86cff000 rlib  01   0  
/usr/lib/libc.so.92.3
031bbc70 031bbc70 ld.so 01   0  
/usr/libexec/ld.so

> Which libraries is "kpasswdd" linked with (post ldd output) and make
> sure you're reporting the results for the right executable.
# ldd /usr/local/heimdal/libexec/kpasswdd
/usr/local/heimdal/libexec/kpasswdd:
StartEnd  Type  Open Ref GrpRef Name
1a995ac0 1a995ae08000 exe   20   0  
/usr/local/heimdal/libexec/kpasswdd
1a9c00c4d000 1a9c00e63000 rlib  01   0  
/usr/local/heimdal/lib/libkadm5srv.so.3.0
1a9b8f90 1a9b8fb04000 rlib  08   0  
/usr/local/lib/libcom_err.so.21.0
1a9ba5981000 1a9ba5c17000 rlib  03   0  
/usr/local/heimdal/lib/libkrb5.so.22.0
1a9b7f5e9000 1a9b7f893000 rlib  06   0  
/usr/local/heimdal/lib/libasn1.so.22.0
1a9b6757d000 1a9b67792000 rlib  08   0  
/usr/local/heimdal/lib/libroken.so.3.0
1a9c23577000 1a9c23784000 rlib  09   0  
/usr/lib/libutil.so.13.0
1a9c1dc73000 1a9c1de9d000 rlib  05   0  
/usr/local/heimdal/lib/libwind.so.3.0
1a9c2b942000 1a9c2bb52000 rlib  06   0  
/usr/local/heimdal/lib/libheimbase.so.2.0
1a9b5f9db000 1a9b5fc2f000 rlib  04   0  
/usr/local/heimdal/lib/libhx509.so.1.0
1a9c2e2b5000 1a9c2e4f7000 rlib  05   0  
/usr/local/heimdal/lib/libhcrypto.so.0.0
1a9c287f4000 1a9c28bd rlib  06   0  
/usr/lib/libcrypto.so.43.1
1a9b66371000 1a9bf000 rlib  04   0  
/usr/local/heimdal/lib/libheimsqlite.so.1.0
1a9bc584c000 1a9bc5a73000 rlib  02   0  
/usr/local/heimdal/lib/libhdb.so.3.0
1a9c40344000 1a9c4054d000 rlib  011   0  
/usr/lib/libpthread.so.25.1
1a9b62ed2000 1a9b631b2000 rlib  01   0  
/usr/lib/libc.so.92.3
1a9c0780 1a9c0780 ld.so 01   0  
/usr/libexec/ld.so

And yes, these are the right executable called by the rc scripts.

> > With "it worked well" I was just referring to the fact that "kadmin
> > -l cpw ..." doesn't work when kpasswdd is down
> 
> That's not possible.  The "kadmin -l cpw" command

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

You've been 23 secs faster to reply than me to correct the email with
the wrong output! :D

So, the correct one has been sent already. About the source code, well
I've tried with both but the upstream didn't compile so I'll stick to
the OpenBSD one from ports.

On Sat, 2018-08-04 at 14:04 -0400, Viktor Dukhovni wrote:
> > On Aug 4, 2018, at 1:43 PM, ASV  wrote:
> > 
> > On Fri, 2018-08-03 at 14:05 -0400, Viktor Dukhovni wrote:
> > > > On Aug 3, 2018, at 1:43 PM, ASV  wrote:
> > > > 
> > > > No, no crashes using "kadmin -l".
> > > 
> > > This shows that "kadmin" and the libkadm5srv.so library work
> > > fine.
> > > 
> > > Which libraries is "kadmin" linked with (post ldd output)?
> > 
> > # ldd /usr/local/heimdal/bin/kadmin
> > /usr/local/heimdal/bin/kadmin:
> 
> OK, thanks.
> 
> > > Which libraries is "kpasswdd" linked with (post ldd output) and
> > > make
> > > sure you're reporting the results for the right executable.
> > 
> > # ldd /usr/local/heimdal/bin/kpasswd
> > /usr/local/heimdal/bin/kpasswd:
> 
> That's the "kpasswd" client, NOT the "kpasswdd" server.  Please post
> the
> "ldd" output for "kpasswdd" making sure to check that the daemon
> that's
> segfaulting is that one, and not some other executable in another
> location.
> 
> The code to build with debugging symbols should ideally be the source
> for
> the installed package, as modified by the OpenBSD maintainers, not
> the
> upstream source (if different).
> 
> -- 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

Good morning,
this looks more exhaustive to me. :)

(gdb) r
Starting program: /usr/local/heimdal/libexec/kpasswdd

Program received signal SIGSEGV, Segmentation fault.
0x1fe263d185d6 in change () from /usr/local/heimdal/lib/libkadm5srv.so.3.0
Current language:  auto; currently minimal
(gdb) bt
#0  0x1fe263d185d6 in change () from 
/usr/local/heimdal/lib/libkadm5srv.so.3.0
#1  0x1fe263d1855f in kadm5_s_chpass_principal_cond () from 
/usr/local/heimdal/lib/libkadm5srv.so.3.0
#2  0x1fe05dc02bfb in change (auth_context=0x1fe261682080, 
admin_principal=0x1fe318614860, version=65408, s=8, sa=0x7f7e0968, 
sa_size=16, in_data=0x7f7e0310) at kpasswdd.c:410
#3  0x1fe05dc01b89 in process (keytab=0x1fe27c7a0c00, s=8, 
this_addr=0x1fe261684330, sa=0x7f7e0968, sa_size=16, msg=0x7f7e0460 
"\002\200\002\bn\202\002\0040\202\002", len=676)
at kpasswdd.c:633
#4  0x1fe05dc017d9 in doit (keytab=0x1fe27c7a0c00, port=53249) at 
kpasswdd.c:767
#5  0x1fe05dc01073 in main (argc=1, argv=0x7f7e0c08) at kpasswdd.c:906
(gdb) x/i $pc
0x1fe263d185d6 :cmpl   $0x0,0xc8(%r14)
(gdb) i reg
rax0xb216510a6421fab2   -5614210780399273294
rbx0x1  1
rcx0x0  0
rdx0x1  1
rsi0x1fe318612740   35060227057472
rdi0x0  0
rbp0x7f7e0050   0x7f7e0050
rsp0x7f7dff40   0x7f7dff40
r8 0x0  0
r9 0x1fe2b9c4c5d0   35058639750608
r100x1fe2d5235b30   35059098934064
r110x0  0
r120x1fe318612740   35060227057472
r130x0  0
r140x0  0
r150x1fe2b9c4c5d0   35058639750608
rip0x1fe263d185d6   0x1fe263d185d6 
eflags 0x10202  66050
cs 0x2b 43
ss 0x23 35
ds 0x23 35
es 0x23 35
fs 0x23 35
gs 0x23 35



On Sat, 2018-08-04 at 15:44 -0400, Viktor Dukhovni wrote:
> > On Aug 4, 2018, at 3:28 PM, ASV  wrote:
> > 
> > I think that I finally got it, did I?
> 
> Still no symbols, but yes, you got the instruction decode.
> 
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x07fe9c69d5d6 in change () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > Current language:  auto; currently minimal
> > (gdb) bt
> > #0  0x07fe9c69d5d6 in change () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > #1  0x07fe9c69d55f in kadm5_s_chpass_principal_cond () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > #2  0x07fbfc702bfb in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #3  0x07fbfc701b89 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #4  0x07fbfc7017d9 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #5  0x07fbfc701073 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #6  0x07fbfc7009a6 in ?? () from
> > /usr/local/heimdal/libexec/kpasswdd
> > #7  0x in ?? ()
> > (gdb) x/i
> > $pc
> >  
> > 0x7fe9c69d5d6 : cmpl   $0x0,0xc8(%r14)
> > (gdb) x/40i $pc-102
> > 0x7fe9c69d570 : push   %rbp
> > 0x7fe9c69d571 :   mov%rsp,%rbp
> > 0x7fe9c69d574 :   push   %r15
> > 0x7fe9c69d576 :   push   %r14
> > 0x7fe9c69d578 :   push   %r13
> > 0x7fe9c69d57a :  push   %r12
> > 0x7fe9c69d57c :  push   %rbx
> > 0x7fe9c69d57d :  sub$0xe8,%rsp
> > 0x7fe9c69d584 :  mov%r9,%r15
> > 0x7fe9c69d587 :  mov%r8,%r13
> > 0x7fe9c69d58a
> > :  mov%ecx,0xff04(%rbp)
> > 0x7fe9c69d590 :  mov%edx,%ebx
> > 0x7fe9c69d592 :  mov%rsi,%r12
> > 0x7fe9c69d595 :  mov%rdi,%r14
> > 0x7fe9c69d598 :  mov2174785(%rip),%rax#
> > 0x7fe9c8b04e0 <__guard_local>
> > 0x7fe9c69d59f
> > :  mov%rax,0xffd0(%rbp)
> > 0x7fe9c69d5a3 :  xorps  %xmm0,%xmm0
> > 0x7fe9c69d5a6 :  movaps
> > %xmm0,0xffc0(%rbp)
> > 0x7fe9c69d5aa :  movaps
> > %xmm0,0xffb0(%rbp)
> > 0x7fe9c69d5ae :  movaps
> > %xmm0,0xffa0(%rbp)
> > 0x7fe9c69d5b2 :  movaps
> > %xmm0,0xff90(%rbp)
> > 0x7fe9c69d5b6 :  movaps
> > %xmm0,0xff80(%rbp)
> > 0x7fe9c69d5ba :  movaps
> > %xmm0,0xff70(%rbp)
> > 0x7fe9c69d5c1 :  movaps
> > %xmm0,0xff60(%rbp)
> > 0x7fe9c69d5c8 :  movaps
> > %xmm0,0xff50(%rbp)
> > 0x7fe9c69d5cf :  movaps
> > %xmm0,0xff40(%rbp)
> > 0x7fe9c69d5d6 : cmpl   $0x0,0xc8(%r14)
> 
> This is confirms the guess, the kadm5 handle is NULL.  We now need
> debugging
> symbols.
> 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

Thank you Antoine,
I'll try and post the results ASAP.

On Sat, 2018-08-04 at 23:01 +0200, Antoine Jacoutot wrote:
> > > Presumably you did not install an executable with the debug
> > > symbols.
> > 
> > About this I do not know what to say, I've seen the CFLAGS in the
> > compiling output lines and I even posted in the email.
> > I will try again as soon as I can. 
> 
> If you build from ports, add INSTALL_STRIP="" to your make command,
> otherwise
> the binaries will be stripped at install time.
> 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

Here we go:

(gdb) frame 2
#2  0x1fe05dc02bfb in change (auth_context=0x1fe261682080, 
admin_principal=0x1fe318614860, version=65408, s=8, sa=0x7f7e0968, 
sa_size=16, in_data=0x7f7e0310) at kpasswdd.c:410
410 ret = kadm5_s_chpass_principal_cond (kadm5_handle, principal, 1, 
tmp);
(gdb) p kadm5_handle
$1 = (void *) 0x0
(gdb) p principal->name.name_string.val[0]
$2 = 0x1fe312b39eb0 "vax"
(gdb) p principal->name.name_string.val[1]
$3 = 0x0
(gdb) p admin_principal->name.name_string[0]
Structure has no component named operator[].
(gdb) p admin_principal->name.name_string[1]
Structure has no component named operator[].
(gdb) p pwd_data->length
$4 = 16
(gdb) p context[0]
Cannot perform pointer math on incomplete type "krb5_context_data", try casting 
to a known type, or void *.



On Sun, 2018-08-05 at 12:11 -0400, Viktor Dukhovni wrote:
> On Aug 5, 2018, at 5:20 AM, ASV  wrote:
> > 
> > Good morning,
> > this looks more exhaustive to me. :)
> 
> You have debugging symbols in the "kpasswdd" executable, but NOT in
> the
> libkadm5srv.so library.  Please also install the re-compiled library
> and
> rerun the test, but first:
> 
> > (gdb) bt
> > #0  0x1fe263d185d6 in change () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > #1  0x1fe263d1855f in kadm5_s_chpass_principal_cond () from
> > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> 
> [ No symbols in frames 0 and 1 ]
> 
> > #2  0x1fe05dc02bfb in change (auth_context=0x1fe261682080,
> > admin_principal=0x1fe318614860, version=65408, s=8,
> > sa=0x7f7e0968, sa_size=16, in_data=0x7f7e0310) at
> > kpasswdd.c:410
> 
> This is the call to kadm5_s_chpass_principal_cond in kpasswdd.c:
> 
> 410 ret = kadm5_s_chpass_principal_cond (kadm5_handle,
> principal, 1, tmp);
> 
> It would already be useful to see the value of "kadm5_handle".  You
> can report the
> output of:
> 
>   (gdb) frame 2
>   (gdb) p kadm5_handle
> (gdb) p principal->name.name_string.val[0]
>   (gdb) p principal->name.name_string.val[1]
>   (gdb) p admin_principal->name.name_string[0]
>   (gdb) p admin_principal->name.name_string[1]
>   (gdb) p pwd_data->length
>   (gdb) p context[0]
> 
> If this proves insufficient, we'll need the debugging symbols for
> library,
> but it is possible that symbols in the library will not be needed.
> 
> > #3  0x1fe05dc01b89 in process (keytab=0x1fe27c7a0c00, s=8,
> > this_addr=0x1fe261684330, sa=0x7f7e0968, sa_size=16,
> > msg=0x7f7e0460 "\002\200\002\bn\202\002\0040\202\002", len=676)
> >at kpasswdd.c:633
> > #4  0x1fe05dc017d9 in doit (keytab=0x1fe27c7a0c00, port=53249)
> > at kpasswdd.c:767
> > #5  0x1fe05dc01073 in main (argc=1, argv=0x7f7e0c08) at
> > kpasswdd.c:906
> > (gdb) x/i $pc
> > 0x1fe263d185d6 :cmpl   $0x0,0xc8(%r14)
> > (gdb) i reg
> > rax0xb216510a6421fab2   -5614210780399273294
> > rbx0x1  1
> > rcx0x0  0
> > rdx0x1  1
> > rsi0x1fe318612740   35060227057472
> > rdi0x0  0
> > rbp0x7f7e0050   0x7f7e0050
> > rsp0x7f7dff40   0x7f7dff40
> > r8 0x0  0
> > r9 0x1fe2b9c4c5d0   35058639750608
> > r100x1fe2d5235b30   35059098934064
> > r110x0  0
> > r120x1fe318612740   35060227057472
> > r130x0  0
> > r140x0  0
> > r150x1fe2b9c4c5d0   35058639750608
> > rip0x1fe263d185d6   0x1fe263d185d6 
> > eflags 0x10202  66050
> > cs 0x2b 43
> > ss 0x23 35
> > ds 0x23 35
> > es 0x23 35
> > fs 0x23 35
> > gs 0x23 35
> > 
> > 
> > 
> > On Sat, 2018-08-04 at 15:44 -0400, Viktor Dukhovni wrote:
> > > > On Aug 4, 2018, at 3:28 PM, ASV  wrote:
> > > > 
> > > > I think that I finally got it, did I?
> > > 
> > > Still no symbols, but yes, you got the instruction decode.
> > > 
> > > > Program received signal SIGSEGV, Segmentation fault.
> > > > 0x07fe9c69d5d6 in change () from
> > > > /usr/local/heimdal/lib/libkadm5srv.so.3.0
> > > > Current language:  auto; currently minimal
> > > > (gdb) bt
> > > > #0  0x07fe9c69d5d6 in change () from
> > > > /usr/local/heimdal/lib/libkadm5

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

Now doesn't crash finally but works only if the principal is not
specified in the kadmind.acl file.

For example I'm changing the password of a...@bla.net and the principle
IS in the acl file as:
a...@bla.netc (or C which should deny it)

the result is
FROM SERVER:
Aug  5 23:47:59 kdc-core-01 kpasswdd[67537]:
kadm5_init_with_password_ctx: Operation failed for unspecified reason

FROM CLIENT:
Hard error : Internal error

If the principal is NOT in the acl file then everything works well:
Aug  5 23:52:17 kdc-core-01 kpasswdd[67537]: Changing password
for a...@bla.net

Doesn't look like an expected behaviour.


On Sun, 2018-08-05 at 16:01 -0400, Viktor Dukhovni wrote:
> > On Aug 5, 2018, at 3:58 PM, ASV  wrote:
> > 
> > Anyway, looking forward for the patch. I'm glad I've helped the
> > project
> > somehow. Thanks a lot for your time and responsiveness.
> 
> https://github.com/heimdal/heimdal/commit/dd249257e397a26c48164122c89
> 2c96a10b64c44
> 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

My kadmind.acl is not tuned at all, I've just given full power to the
users listed using different forms for testing as this is not a
production environment yet.

According to the manual:
* is the same as x
x is a short for admcilsp. All privileges (except e)

vaxx...@bla.net *
vaxx...@bla.net e
vax/ad...@bla.net   admcilsp
root/ad...@bla.net   admcilsp
root/ad...@bla.net   e
root/ad...@bla.net   *

I realized that I've specified erroneously root/admin with literally
most of the permissions duplicated. It is certainly an ugly mistake,
"debris" forgotten due to the huge amont of tests I did on this KDC.
Nevertheless I would expect some complaining in the log files instead
of a daemon to die.

Anyway, looking forward for the patch. I'm glad I've helped the project
somehow. Thanks a lot for your time and responsiveness.




On Sun, 2018-08-05 at 14:15 -0400, Viktor Dukhovni wrote:
> > On Aug 5, 2018, at 12:33 PM, ASV  wrote:
> > 
> > Here we go:
> > 
> > (gdb) frame 2
> > #2  0x1fe05dc02bfb in change (auth_context=0x1fe261682080,
> > admin_principal=0x1fe318614860, version=65408, s=8,
> > sa=0x7f7e0968, sa_size=16, in_data=0x7f7e0310) at
> > kpasswdd.c:410
> > 410 ret = kadm5_s_chpass_principal_cond (kadm5_handle,
> > principal, 1, tmp);
> > (gdb) p kadm5_handle
> > $1 = (void *) 0x0
> 
> OK, so no need for a debugging version of the libkadm5srv.so library,
> ...
> Passing it a NULL kadm5_handle is expected to segfault.  The kpasswdd
> program should not be doing that!
> 
> So the question is how the handle ended up being NULL without
> kpasswdd
> bailing out earlier.
> 
> The function that creates the handle is
> kadm5_init_with_password_ctx(),
> which after a bit of indirection ends up in
> kadm5_s_init_with_context():
> 
>  99 ret = krb5_parse_name(ctx->context, client_name, 
> >caller);
> 100 if (ret == 0)
> 101 ret = _kadm5_acl_init(ctx);
> 102 if (ret)
> 103 kadm5_s_destroy(ctx);
> 104 else
> 105 *server_handle = ctx;
> 106 return 0;
> 
> Line 106 of that function is incorrect.  It should be "return ret",
> NOT "return 0".  If either krb5_parse_name() or _kadm5_acl_init()
> fail, you'll end up with a NULL kadm5 handle, and a success (0)
> return code.
> 
> Most likely there's some sort of problem with your admin ACL file.
> And we need to fix line 106 of "lib/kadm5/init_s.c".
> 
> Thanks for the problem report.
> 
> -- 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

Awesome! I'll modify it manually as it's gonna take some time to get
into the OpenBSD ports anyway.

Thanks.

On Sun, 2018-08-05 at 16:01 -0400, Viktor Dukhovni wrote:
> https://github.com/heimdal/heimdal/commit/dd249257e397a26c48164122c89
> 2c96a10b64c44

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

I'm not really mastering Heimdal just yet so I'd like to be sure I'm
not wrong. But for instance the iprop configuration guide IMO is
incomplete.
https://www.h5l.org/manual/HEAD/info/heimdal/Incremental-propagation.ht
ml

"Then you need to create those principals that you added in the
configuration file. Create one `iprop/hostname' for the master and for
every slave.

 master# /usr/heimdal/sbin/ktutil get iprop/`hostname`

 slave# /usr/heimdal/sbin/ktutil get iprop/`hostname`"


How is the slave supposed to connect to the master if has a principal
built on the server itself which is not on the master?
In fact following the guide the result, when trying to connect the
slave to the master is something like:
"iprop/slave-host principal not in hdb".

I've fixed it creating an iprop/slave-host on the master.

Another thing in this regard, which I don't know why is it behaving
like that, but I cannot somehow secure such principal which means that
I cannot set a password because it will always fail.
Therefore the principal has to be with no password leaving the access
control to the check in /var/heimdal/slaves and the identification of
the FQDN declared in the principal itself.

Am I wrong?



On Mon, 2018-08-06 at 10:57 -0400, Viktor Dukhovni wrote:
> > On Aug 6, 2018, at 3:52 AM, ASV  wrote:
> > 
> > There are sections which are scarcely written and perhaps not even
> > correct (like the incremental propagation one).
> 
> If you could be a bit more specific, (and as Jeff says open an
> issue on Github) we might be able to address some of the most
> critical deficits.
> 

Re: kpasswdd dumps on OpenBSD6.3

[ASV]

You're right, I've been reading a lot the MIT documentation because the
Heimdal one is not very good. I made the wrong assumption that also the
acl file syntax was the same, sorry.
By the way, Heimdal is a pretty old project at this point in time but
it doesn't seem to be very actively maintained. Is there a way to
improve the documentation somehow?
There are sections which are scarcely written and perhaps not even
correct (like the incremental propagation one).

On Sun, 2018-08-05 at 18:15 -0400, Viktor Dukhovni wrote:
> > On Aug 5, 2018, at 5:58 PM, ASV  wrote:
> > 
> > For example I'm changing the password of a...@bla.net and the
> > principle
> > IS in the acl file as:
> > a...@bla.netc (or C which should deny it)
> 
> You're reading the MIT Kerberos documentation for the kadmind.acl
> file.  In heimdal the syntax is different.  See for example:
> 
>https://manpages.debian.org/jessie/heimdal-kdc/kadmind.8.en.html
>