Re: Random and rare Seg faults at openssl library level

[Matt Caswell]

On 12/01/2021 04:23, Gimhani Uthpala wrote:
> Hi team,
> https://www.openssl.org/docs/man1.0.2/man3/CRYPTO_set_locking_callback.html
> :  From this , learnt that for openssl multi-threaded applications to be
> run safely, the callback functions to be implemented.
> 
> I am using this in a multi-threaded application and Above scenario was
> again recreated and in that case 2 threads were calling RSA_verify (same
> stack above). However, for SSL_do_handshake, SSL* objects are given
> separately and they are not shared between threads. So, we have not
> implemented callback functions.
> 
> Could you please clarify whether we need to implement locking callbacks
> for multi-threaded applications  even if the ssl objects we supply are
> provided separately for threads?

If you are using 1.0.2 in a multi-threaded context then you *must*
implement the locking callbacks. There are resources that are shared
between multiple threads that require appropriate locking.

Matt



> 
> Many thanks.
> 
> On Wed, Jan 6, 2021 at 10:40 PM Gimhani Uthpala
> mailto:gimhanieuthp...@gmail.com>> wrote:
> 
> Dear team,
> I'm running an application which uses openssl for secure
> communication between processes. I am getting seg-faults at openssl
> level. This only occurred very randomly and the following are stacks
> that seg faults  at openssl level in the given 2 cases. We are using
> openssl 1.0.2k. 
> 
> Went through the security vulnerabilities list for this version but
> couldn't find a clue. Running valgrind too didn't give an exact clue
> related to the issue. Can you please guide me how can I find the
> exact root cause for the seg fault?  
> 
> I am calling SSL_do_handshake(ssl_ctx) from my code level and both
> the below seg faults are occuring from it's inside. 
> 
> #0  0x7fd64cdabdd3 in ASN1_item_verify () from
> /lib64/libcrypto.so.10
> #1  0x7fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
> #2  0x7fd64cdccaef in X509_verify_cert () from
> /lib64/libcrypto.so.10
> #3  0x7fd64d111c68 in ssl_verify_cert_chain () from
> /lib64/libssl.so.10
> #4  0x7fd64d0e8cc6 in ssl3_get_client_certificate () from
> /lib64/libssl.so.10
> *#5  0x7fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10*
> 
> 
>  #0 0x7ffb65529854 in RSA_verify () from /usr/lib64/libcrypto.so.10
>  #1 0x7ffb6552f1fc in pkey_rsa_verify () from
> /usr/lib64/libcrypto.so.10
>  #2 0x7ffb65561877 in EVP_DigestVerifyFinal () from
> /usr/lib64/libcrypto.so.10
>  #3 0x7ffb6556af25 in ASN1_item_verify () from
> /usr/lib64/libcrypto.so.10
>  #4 0x7ffb65589c58 in internal_verify () from
> /usr/lib64/libcrypto.so.10
>  #5 0x7ffb6558baef in X509_verify_cert () from
> /usr/lib64/libcrypto.so.10
>  #6 0x7ffb658d1417 in ssl_add_cert_chain () from
> /usr/lib64/libssl.so.10
>  #7 0x7ffb658b6095 in ssl3_output_cert_chain () from
> /usr/lib64/libssl.so.10
>  #8 0x7ffb658ae894 in ssl3_send_client_certificate () from
> /usr/lib64/libssl.so.10
> * #9 0x7ffb658aeecf in ssl3_connect () from /usr/lib64/libssl.so.10
>  #10 0x7ffb658b8e7e in ssl23_connect () from
> /usr/lib64/libssl.so.10*
> 
>   I am setting context to use SSLv23_method() s. However, I can see
> ssl3_ methods being called. Is there any issue with that?  
> 
> Given below is SSL* object passed to SSL_do_handshake method.
> 
> (gdb) p *p_SSLCtx
> $2 = {version = 771, type = 4096, method = 0x7ffb65af3320, rbio =
> 0x7ffb5819e140, wbio = 0x7ffb58193570, bbio = 0x7ffb58193570,
> rwstate = 1, in_handshake = 2, *handshake_func = 0x7ffb658aea30
> *, server = 0, new_session = 0, quiet_shutdown = 0,
> shutdown = 0, state = 4467,
>   rstate = 240, init_buf = 0x7ffb581934b0, init_msg =
> 0x7ffb581e10d4, init_num = 0, init_off = 0, *packet = 0x7ffb581e6633
> "\026\003\003", packet_length = 0*, s2 = 0x0, s3 = 0x7ffb58195ee0,
> d1 = 0x0, read_ahead = 0, msg_callback = 0x0, msg_callback_arg =
> 0x0, hit = 0, param = 0x7ffb581933f0,
>   cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0,
> enc_read_ctx = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx =
> 0x0, write_hash = 0x0, compress = 0x0, cert = 0x7ffb58195ba0,
> sid_ctx_length = 0, sid_ctx = '\000' , session =
> 0x7ffb58198100,
>   generate_session_id = 0x0,..
>  
> 
> 
> -- 
> *Gimhani Uthpala Kankanamge*
> 
> 
> 
> -- 
> *Gimhani Uthpala Kankanamge*

Re: Random and rare Seg faults at openssl library level

[Jan Just Keijser]

Hi,

On 07/01/21 23:53, Gimhani Uthpala wrote:



On Thu, Jan 7, 2021 at 3:08 AM Ken Goldman > wrote:


On 1/6/2021 12:10 PM, Gimhani Uthpala wrote:

> I am getting seg-faults at openssl level. This only
occurred very randomly and the following are stacks that seg
faults  at openssl level in the given 2 cases. We are using
openssl 1.0.2k.

The usual cause is that you are compiling with one version of
openssl and (static or dynamic) linking with a different one.
The cause of that is typically that you have more than one version
of openssl installed.

If this is a 3rd party application, not one you're building, you
have to find out what version of openssl they expect.



I only have this 1.0.2.k-fips one version installed in both compiling 
and running machines. However, I am compiling the application in RH7.4 
and running in RH7.8 linking to openssl library dynamically. I assume 
no issue with that as I am using the same version of openssl in both.


actually - there could be an issue with that, as RedHat has this 
tendency to patch openssl between releases (mostly backporting security 
fixes from openssl 1.1.x to 1.0.2k).


Have you tried installing the debuginfo package for openssl so that the 
stacktrace will show you better info:

  debuginfo-install openssl-libs openssl

right now all you know is that the segfault occurs *somewhere* within 
ASN1_item_verify ()


HTH,

JJK

Re: Random and rare Seg faults at openssl library level

[Gimhani Uthpala]

Hi team,
https://www.openssl.org/docs/man1.0.2/man3/CRYPTO_set_locking_callback.html
:  From this , learnt that for openssl multi-threaded applications to be
run safely, the callback functions to be implemented.

I am using this in a multi-threaded application and Above scenario was
again recreated and in that case 2 threads were calling RSA_verify (same
stack above). However, for SSL_do_handshake, SSL* objects are given
separately and they are not shared between threads. So, we have not
implemented callback functions.

Could you please clarify whether we need to implement locking callbacks for
multi-threaded applications  even if the ssl objects we supply are provided
separately for threads?

Many thanks.

On Wed, Jan 6, 2021 at 10:40 PM Gimhani Uthpala 
wrote:

> Dear team,
> I'm running an application which uses openssl for secure communication
> between processes. I am getting seg-faults at openssl level. This only
> occurred very randomly and the following are stacks that seg faults  at
> openssl level in the given 2 cases. We are using openssl 1.0.2k.
>
> Went through the security vulnerabilities list for this version but
> couldn't find a clue. Running valgrind too didn't give an exact clue
> related to the issue. Can you please guide me how can I find the exact root
> cause for the seg fault?
>
> I am calling SSL_do_handshake(ssl_ctx) from my code level and both the
> below seg faults are occuring from it's inside.
>
> #0  0x7fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10
> #1  0x7fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
> #2  0x7fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10
> #3  0x7fd64d111c68 in ssl_verify_cert_chain () from /lib64/libssl.so.10
> #4  0x7fd64d0e8cc6 in ssl3_get_client_certificate () from
> /lib64/libssl.so.10
> *#5  0x7fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10*
>
>
>  #0 0x7ffb65529854 in RSA_verify () from /usr/lib64/libcrypto.so.10
>  #1 0x7ffb6552f1fc in pkey_rsa_verify () from
> /usr/lib64/libcrypto.so.10
>  #2 0x7ffb65561877 in EVP_DigestVerifyFinal () from
> /usr/lib64/libcrypto.so.10
>  #3 0x7ffb6556af25 in ASN1_item_verify () from
> /usr/lib64/libcrypto.so.10
>  #4 0x7ffb65589c58 in internal_verify () from
> /usr/lib64/libcrypto.so.10
>  #5 0x7ffb6558baef in X509_verify_cert () from
> /usr/lib64/libcrypto.so.10
>  #6 0x7ffb658d1417 in ssl_add_cert_chain () from
> /usr/lib64/libssl.so.10
>  #7 0x7ffb658b6095 in ssl3_output_cert_chain () from
> /usr/lib64/libssl.so.10
>  #8 0x7ffb658ae894 in ssl3_send_client_certificate () from
> /usr/lib64/libssl.so.10
>
> * #9 0x7ffb658aeecf in ssl3_connect () from
> /usr/lib64/libssl.so.10 #10 0x7ffb658b8e7e in ssl23_connect () from
> /usr/lib64/libssl.so.10*
>
>   I am setting context to use SSLv23_method() s. However, I can see ssl3_
> methods being called. Is there any issue with that?
>
> Given below is SSL* object passed to SSL_do_handshake method.
>
> (gdb) p *p_SSLCtx
> $2 = {version = 771, type = 4096, method = 0x7ffb65af3320, rbio =
> 0x7ffb5819e140, wbio = 0x7ffb58193570, bbio = 0x7ffb58193570, rwstate = 1,
> in_handshake = 2, *handshake_func = 0x7ffb658aea30 *,
> server = 0, new_session = 0, quiet_shutdown = 0, shutdown = 0, state = 4467,
>   rstate = 240, init_buf = 0x7ffb581934b0, init_msg = 0x7ffb581e10d4,
> init_num = 0, init_off = 0, *packet = 0x7ffb581e6633 "\026\003\003",
> packet_length = 0*, s2 = 0x0, s3 = 0x7ffb58195ee0, d1 = 0x0, read_ahead =
> 0, msg_callback = 0x0, msg_callback_arg = 0x0, hit = 0, param =
> 0x7ffb581933f0,
>   cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0, enc_read_ctx
> = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx = 0x0, write_hash =
> 0x0, compress = 0x0, cert = 0x7ffb58195ba0, sid_ctx_length = 0, sid_ctx =
> '\000' , session = 0x7ffb58198100,
>   generate_session_id = 0x0,..
>
>
>
> --
> *Gimhani Uthpala Kankanamge*
>
>

-- 
*Gimhani Uthpala Kankanamge*

Re: Random and rare Seg faults at openssl library level

[Jakob Bohm via openssl-users]

On 2021-01-07 18:05, Ken Goldman wrote:

On 1/7/2021 10:11 AM, Michael Wojcik wrote:


$ cat /etc/redhat-release && openssl version
CentOS Linux release 7.9.2009 (Core)
OpenSSL 1.0.2k-fips  26 Jan 2017


Ugh. Well, OP should have made that clear in the original message.

And this is one of the problems with using an OpenSSL supplied by the 
OS vendor.


In defense of "the OS vendor", meaning the distro, it's a big task to
upgrade to a new openssl major release.  Because there is often not ABI
compatibility, every package has to be ported, built, and tested.
A distro release that is in long term support doesn't do that often.




In defense of long term support distros, until a few years ago, no one 
suspected that OpenSSL would come under a new leadership that actively 
did everything to make it near-impossible to maintain backported 
security patches for a typical 5+ year distro lifecycle (with 
OpenSSL-independent start date).


Until 1.0.2, all OpenSSL releases were incremental patch-steps from the 
old 0.9.x series, allowing distro maintainers to manually cherry pick 
changes for doing ABI-compatible patches for whichever 1.0.x or 0.9.x 
was current at the start of their lifecycle.  Then the new leadership 
started to restructure the code even in supposedly patch-level releases.


A lot of long term support distros are now firmly stuck with unsupported 
OpenSSL 1.0.2 and/or short life cycle 1.1.1.


Not all long term distros are run by rich companies like IBM/RedHat that 
can purchase support plans, resulting in further popularity of OpenSSL 
forks.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

RE: Random and rare Seg faults at openssl library level

[Kenneth Goldman]

From:   Gimhani Uthpala 
To: Ken Goldman 
Cc: openssl-users@openssl.org
Date:   01/07/2021 05:53 PM
Subject:[EXTERNAL] Re: Random and rare Seg faults at openssl library
level



I only have this 1.0.2.k-fips one version installed in both compiling and
running machines. However, I am compiling the application in RH7.4 and
running in RH7.8 linking to openssl library dynamically. I assume no issue
with that as I am using the same version of openssl in both.

You are having a problem, and that is a typical cause.  Try compiling and
running on the exact same OS.

If you installed openssl yourself, not using the RHEL yum installer, I
would expect random and rare issues.

Re: Random and rare Seg faults at openssl library level

[Gimhani Uthpala]

On Thu, Jan 7, 2021 at 3:08 AM Ken Goldman  wrote:

> On 1/6/2021 12:10 PM, Gimhani Uthpala wrote:
>
> > I am getting seg-faults at openssl level. This only occurred very
> randomly and the following are stacks that seg faults  at openssl level in
> the given 2 cases. We are using openssl 1.0.2k.
>
> The usual cause is that you are compiling with one version of openssl and
> (static or dynamic) linking with a different one.
> The cause of that is typically that you have more than one version of
> openssl installed.
>
> If this is a 3rd party application, not one you're building, you have to
> find out what version of openssl they expect.
>
>
>
I only have this 1.0.2.k-fips one version installed in both compiling and
running machines. However, I am compiling the application in RH7.4 and
running in RH7.8 linking to openssl library dynamically. I assume no issue
with that as I am using the same version of openssl in both.

-Gimhani

Re: Random and rare Seg faults at openssl library level

[Gimhani Uthpala]

On Thu, Jan 7, 2021 at 1:53 PM Jan Just Keijser  wrote:

> On 06/01/21 21:57, Michael Wojcik wrote:
> >
> > The same way you'd track down an intermittent cause of Undefined
> Behavior in any other program: some combination of dynamic monitoring,
> symbolic execution, static code analysis, source code review, testing
> variants, tracing, fuzzing, post-mortem analysis, and so on. This isn't
> specific to OpenSSL.
> >
> > But you're asking the wrong question. The correct question is: Why are
> you using an outdated version of OpenSSL?
>
> possibly because:
>
> $ cat /etc/redhat-release && openssl version
> CentOS Linux release 7.9.2009 (Core)
> OpenSSL 1.0.2k-fips  26 Jan 2017
>
> ?
>
>
>
Yes, using this openssl version coming with the OS.

Re: Random and rare Seg faults at openssl library level

[Gimhani Uthpala]

On Thu, Jan 7, 2021 at 1:51 PM Jan Just Keijser  wrote:

> Hi,
>
> On 06/01/21 18:10, Gimhani Uthpala wrote:
>
> Dear team,
> I'm running an application which uses openssl for secure communication
> between processes. I am getting seg-faults at openssl level. This only
> occurred very randomly and the following are stacks that seg faults  at
> openssl level in the given 2 cases. We are using openssl 1.0.2k.
>
> version 1.0.2k suggests you are using RHEL7/CentOS 7, correct?
>

Yes, I am using RHEL7 and using its openssl version 1.0.2k-fips.


> Went through the security vulnerabilities list for this version but
> couldn't find a clue. Running valgrind too didn't give an exact clue
> related to the issue. Can you please guide me how can I find the exact root
> cause for the seg fault?
>
> I am calling SSL_do_handshake(ssl_ctx) from my code level and both the
> below seg faults are occuring from it's inside.
>
> #0  0x7fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10
> #1  0x7fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
> #2  0x7fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10
> #3  0x7fd64d111c68 in ssl_verify_cert_chain () from /lib64/libssl.so.10
> #4  0x7fd64d0e8cc6 in ssl3_get_client_certificate () from
> /lib64/libssl.so.10
> *#5  0x7fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10*
>
>
> so the segfault occurs inside ASN1_item_verify () when verifying the
> certificate - it could be a malformed certificate with invalid ASN1
> encoding; do you have the certificate that causes the segfault?
>

> If you do not, then it is worthwhile recording/storing all certificates
> until you find the one that causes the segfault and then examine it.
>

I do not have access to the certificate that caused segfault. Will try to
record all certs to check this, Thanks.


>
>
> HTH,
>
> JJK
>

Re: Random and rare Seg faults at openssl library level

[Ken Goldman]

On 1/7/2021 10:11 AM, Michael Wojcik wrote:


$ cat /etc/redhat-release && openssl version
CentOS Linux release 7.9.2009 (Core)
OpenSSL 1.0.2k-fips  26 Jan 2017


Ugh. Well, OP should have made that clear in the original message.

And this is one of the problems with using an OpenSSL supplied by the OS vendor.


In defense of "the OS vendor", meaning the distro, it's a big task to
upgrade to a new openssl major release.  Because there is often not ABI
compatibility, every package has to be ported, built, and tested.
A distro release that is in long term support doesn't do that often.

RE: Random and rare Seg faults at openssl library level

[Michael Wojcik]

> From: Jan Just Keijser 
> Sent: Thursday, 7 January, 2021 01:23
>
> On 06/01/21 21:57, Michael Wojcik wrote:
> >
> >
> > But you're asking the wrong question. The correct question is: Why are you
> > using an outdated version of OpenSSL?
>
> possibly because:
>
> $ cat /etc/redhat-release && openssl version
> CentOS Linux release 7.9.2009 (Core)
> OpenSSL 1.0.2k-fips  26 Jan 2017

Ugh. Well, OP should have made that clear in the original message.

And this is one of the problems with using an OpenSSL supplied by the OS vendor.

--
Michael Wojcik

Re: Random and rare Seg faults at openssl library level

[Matt Caswell]

On 06/01/2021 17:10, Gimhani Uthpala wrote:
>   I am setting context to use SSLv23_method() s. However, I can see
> ssl3_ methods being called. Is there any issue with that?  

Just answering this one side question: no, this is normal behaviour.

Matt

Re: Random and rare Seg faults at openssl library level

[John Wasilewski]

Please remove my email

> On Jan 7, 2021, at 3:23 AM, Jan Just Keijser  wrote:
> 
> On 06/01/21 21:57, Michael Wojcik wrote:
>> 
>> The same way you'd track down an intermittent cause of Undefined Behavior in 
>> any other program: some combination of dynamic monitoring, symbolic 
>> execution, static code analysis, source code review, testing variants, 
>> tracing, fuzzing, post-mortem analysis, and so on. This isn't specific to 
>> OpenSSL.
>> 
>> But you're asking the wrong question. The correct question is: Why are you 
>> using an outdated version of OpenSSL?
> 
> possibly because:
> 
> $ cat /etc/redhat-release && openssl version
> CentOS Linux release 7.9.2009 (Core)
> OpenSSL 1.0.2k-fips  26 Jan 2017
> 
> ?
> 
> 

Re: Random and rare Seg faults at openssl library level

[Jan Just Keijser]

On 06/01/21 21:57, Michael Wojcik wrote:


The same way you'd track down an intermittent cause of Undefined Behavior in 
any other program: some combination of dynamic monitoring, symbolic execution, 
static code analysis, source code review, testing variants, tracing, fuzzing, 
post-mortem analysis, and so on. This isn't specific to OpenSSL.

But you're asking the wrong question. The correct question is: Why are you 
using an outdated version of OpenSSL?


possibly because:

$ cat /etc/redhat-release && openssl version
CentOS Linux release 7.9.2009 (Core)
OpenSSL 1.0.2k-fips  26 Jan 2017

?

Re: Random and rare Seg faults at openssl library level

[Jan Just Keijser]

Hi,

On 06/01/21 18:10, Gimhani Uthpala wrote:

Dear team,
I'm running an application which uses openssl for secure communication 
between processes. I am getting seg-faults at openssl level. This only 
occurred very randomly and the following are stacks that seg faults  
at openssl level in the given 2 cases. We are using openssl 1.0.2k.



version 1.0.2k suggests you are using RHEL7/CentOS 7, correct?
Went through the security vulnerabilities list for this version but 
couldn't find a clue. Running valgrind too didn't give an exact clue 
related to the issue. Can you please guide me how can I find the exact 
root cause for the seg fault?


I am calling SSL_do_handshake(ssl_ctx) from my code level and both the 
below seg faults are occuring from it's inside.


#0  0x7fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10
#1  0x7fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
#2  0x7fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10
#3  0x7fd64d111c68 in ssl_verify_cert_chain () from 
/lib64/libssl.so.10
#4  0x7fd64d0e8cc6 in ssl3_get_client_certificate () from 
/lib64/libssl.so.10

*#5  0x7fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10*


so the segfault occurs inside ASN1_item_verify () when verifying the 
certificate - it could be a malformed certificate with invalid ASN1 
encoding; do you have the certificate that causes the segfault?


If you do not, then it is worthwhile recording/storing all certificates 
until you find the one that causes the segfault and then examine it.



HTH,

JJK

Re: Random and rare Seg faults at openssl library level

[Ken Goldman]

On 1/6/2021 12:10 PM, Gimhani Uthpala wrote:


I am getting seg-faults at openssl level. This only occurred very randomly and 
the following are stacks that seg faults  at openssl level in the given 2 
cases. We are using openssl 1.0.2k.


The usual cause is that you are compiling with one version of openssl and 
(static or dynamic) linking with a different one.
The cause of that is typically that you have more than one version of openssl 
installed.

If this is a 3rd party application, not one you're building, you have to find 
out what version of openssl they expect.

Re: Random and rare Seg faults at openssl library level

[tincanteksup]

On 06/01/2021 20:57, Michael Wojcik wrote:



But you're asking the wrong question. The correct question is: Why are you 
using an outdated version of OpenSSL?

--
Michael Wojcik



:whip-crack: !

RE: Random and rare Seg faults at openssl library level

[Michael Wojcik]

> From: openssl-users  On Behalf Of Gimhani 
> Uthpala
> Sent: Wednesday, 6 January, 2021 10:10

> I'm running an application which uses openssl for secure communication between
> processes. I am getting seg-faults at openssl level. This only occurred very
> randomly and the following are stacks that seg faults  at openssl level in the
> given 2 cases.

> We are using openssl 1.0.2k.

Sometimes you see a question that nearly answers itself.

You're using a release that's approaching four years old, and which is 
unsupported, unless you have a premium support contract from openssl.org or 
similar through another vendor. If you do, that's whom you should ask.

In any case, why are you using 1.0.2k? At the very least you should be using 
the final 1.0.2 release -- and then only if you absolutely can't move to 1.1.1 
(generally because you need FIPS validation, but you don't mention FIPS). And 
then you need a premium support contract, if this is a commercial product. 
Particularly these days it's very hard to forgive a commercial-software vendor 
using an outdated, unsupported third-party component.

The most recent version of 1.0.2 that I happen to have lying around is 1.0.2n, 
and there's nothing in the changelog between 1.0.2k and 1.0.2n which looks 
likely to cause this particular problem (though CVE-2017-3735 is a slight 
contender). But that just means the cause isn't anything obvious between k and 
n.

> Went through the security vulnerabilities list for this version but couldn't
> find a clue. Running valgrind too didn't give an exact clue related to the 
> issue.
> Can you please guide me how can I find the exact root cause for the seg fault?

The same way you'd track down an intermittent cause of Undefined Behavior in 
any other program: some combination of dynamic monitoring, symbolic execution, 
static code analysis, source code review, testing variants, tracing, fuzzing, 
post-mortem analysis, and so on. This isn't specific to OpenSSL.

But you're asking the wrong question. The correct question is: Why are you 
using an outdated version of OpenSSL?

--
Michael Wojcik