I think I know what's the problem. If you return 1 in rsa_keygen, OpenSSL
expects a correct RSA key. I find out muy PKCKS#11 device don't work ok and
the keys was wrong so OpenSSL didn't recive a correct RSA key and when I
returned 1 withouh generate an RSA key was the same problem, OpenSSL hadn't
a RSA key.
El 24 de septiembre de 2010 10:31, Nacho Álvarez <nasin...@gmail.com>escribió:
> I have the engine in a dynamic library and there's not relevant part of the
> code. If I overwrite rsa_keygen function and if I do something or nothing
> but return 1 I get "segmentation fault" but it's not my function, my
> function ends ok and invokes the return instruction. The example code I
> wrote is an example that makes OpenSSL ends with "segmentation fault".
> I don't know how to get gdb output from a dynamic library or OpenSSL.
>
> 2010/9/23 Christian Hohnstaedt <christ...@hohnstaedt.de>
>
> On Thu, Sep 23, 2010 at 10:08:40AM +0200, Nacho ?lvarez wrote:
>> > That's not the problem, I think, because if the only instruction of the
>> > function is:
>> >
>> > int rsa_keygen (RSA *rsa, int bits, BIGNUM *e, BN_GENCB *cb) {
>> > return 1;
>> > }
>> >
>> > I get the "segmentation fault" when the rsa_keygen ends. But if I return
>> 0
>> > or -1 (errors) OpenSSL reports me "error in genrsa" and call ends ok (no
>> > segmentation fault or something like that).
>>
>> What about posting the relevant part of the code?
>> And the gdb output.
>>
>> It would help a lot.
>>
>> Cheers
>>
>> Christian
>>
>>
>> >
>> > 2010/9/22 Christian Hohnstaedt <christ...@hohnstaedt.de>
>> >
>> > > On Wed, Sep 22, 2010 at 03:41:30PM +0200, Nacho ?lvarez wrote:
>> > > > Hello everybody
>> > > >
>> > > > Several months ago I developed an OpenSSL PKCS#11 engine for Windows
>> XP
>> > > and
>> > > > it worked ok (it was compiled with MinGW).
>> > > > Now I'm trying to compile it on Linux (Debian 5 with GCC 4.3.1) but
>> I
>> > > have
>> > > > the next problem:
>> > > >
>> > > > In the overwritten function *rsa_keygen* (whose signature is *int
>> > > rsa_keygen
>> > > > (RSA *rsa, int bits, BIGNUM *e, BN_GENCB *cb)*) always when the
>> function
>> > > > ends (after the return instrucction) System reports me *Segmentation
>> > > Fault*.
>> > >
>> > > You overwrote you return address on the stack most probably by
>> > > accessing a locally declared array out of bounds.
>> > >
>> > > Cheers
>> > >
>> > > Christian
>> > > ______________________________________________________________________
>> > > OpenSSL Project
>> http://www.openssl.org
>> > > User Support Mailing List
>> openssl-users@openssl.org
>> > > Automated List Manager
>> majord...@openssl.org
>> > >
>> ______________________________________________________________________
>> OpenSSL Project http://www.openssl.org
>> User Support Mailing List openssl-users@openssl.org
>> Automated List Manager majord...@openssl.org
>>
>
>
