I have the engine in a dynamic library and there's not relevant part of the
code. If I overwrite rsa_keygen function and if I do something or nothing
but return 1 I get "segmentation fault" but it's not my function, my
function ends ok and invokes the return instruction. The example code I
wrote is an example that makes OpenSSL ends with "segmentation fault".
I don't know how to get gdb output from a dynamic library or OpenSSL.
2010/9/23 Christian Hohnstaedt <christ...@hohnstaedt.de>
> On Thu, Sep 23, 2010 at 10:08:40AM +0200, Nacho ?lvarez wrote:
> > That's not the problem, I think, because if the only instruction of the
> > function is:
> >
> > int rsa_keygen (RSA *rsa, int bits, BIGNUM *e, BN_GENCB *cb) {
> > return 1;
> > }
> >
> > I get the "segmentation fault" when the rsa_keygen ends. But if I return
> 0
> > or -1 (errors) OpenSSL reports me "error in genrsa" and call ends ok (no
> > segmentation fault or something like that).
>
> What about posting the relevant part of the code?
> And the gdb output.
>
> It would help a lot.
>
> Cheers
>
> Christian
>
>
> >
> > 2010/9/22 Christian Hohnstaedt <christ...@hohnstaedt.de>
> >
> > > On Wed, Sep 22, 2010 at 03:41:30PM +0200, Nacho ?lvarez wrote:
> > > > Hello everybody
> > > >
> > > > Several months ago I developed an OpenSSL PKCS#11 engine for Windows
> XP
> > > and
> > > > it worked ok (it was compiled with MinGW).
> > > > Now I'm trying to compile it on Linux (Debian 5 with GCC 4.3.1) but I
> > > have
> > > > the next problem:
> > > >
> > > > In the overwritten function *rsa_keygen* (whose signature is *int
> > > rsa_keygen
> > > > (RSA *rsa, int bits, BIGNUM *e, BN_GENCB *cb)*) always when the
> function
> > > > ends (after the return instrucction) System reports me *Segmentation
> > > Fault*.
> > >
> > > You overwrote you return address on the stack most probably by
> > > accessing a locally declared array out of bounds.
> > >
> > > Cheers
> > >
> > > Christian
> > > ______________________________________________________________________
> > > OpenSSL Project http://www.openssl.org
> > > User Support Mailing List openssl-users@openssl.org
> > > Automated List Manager majord...@openssl.org
> > >
> ______________________________________________________________________
> OpenSSL Project http://www.openssl.org
> User Support Mailing List openssl-users@openssl.org
> Automated List Manager majord...@openssl.org
>
