memory managemtn with openssl

2007-07-19 Thread Edward Chan
Hi there,

 

Just curious if anybody has done any profiling of openssl's memory
usage?  Is there much heap contention?  Has anybody tried plugging in a
3rd party memory manager such as Hoard, or SmartHeap to see if there is
any performance improvement?

 

Thanks,

Ed



Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Fox


On Jul 19, 2007, at 9:45 PM, Jim Marshall wrote:


Jim Marshall wrote:

David Schwartz wrote:
OpenSSL tries to make SSL connections act like regular TCP  
connections. This
is exactly what TCP does. So your application would have this  
exact same
problem with or without OpenSSL. As such, how can you blame it on  
OpenSSL?
Not to beat a dead horse, but I forgot to mention that the  
application does work "properly" when performing the same  
operations on non-SSL connections. In other-words if I use telnet  
to connect to the server on the non-SSL port and type nothing in  
the console and then have a second client connect (to either the  
non-SSL or the SSL port) and make a valid request, the server  
responds properly. The telnet session is still connected, but we  
are not blocked from handling other requests. The WEBs code uses  
the socket 'select' mechanism which I presume plays a role in this  
working. I do not understand why this same mechanism fails when we  
use SSL.




The 'same mechanism' is not failing when you use openssl.  You're  
just not doing it right.  If you set your SSL sockets to non-blocking  
and include any SSL socket 'want-read' and 'want-write' in your  
selects everything will work.   There is no magic bullet though.  SSL  
is more complex than telnet and you have to deal with that complexity.


Jim
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Marshall



Jim Marshall wrote:

David Schwartz wrote:
OpenSSL tries to make SSL connections act like regular TCP 
connections. This

is exactly what TCP does. So your application would have this exact same
problem with or without OpenSSL. As such, how can you blame it on 
OpenSSL?
Not to beat a dead horse, but I forgot to mention that the application 
does work "properly" when performing the same operations on non-SSL 
connections. In other-words if I use telnet to connect to the server on 
the non-SSL port and type nothing in the console and then have a second 
client connect (to either the non-SSL or the SSL port) and make a valid 
request, the server responds properly. The telnet session is still 
connected, but we are not blocked from handling other requests. The WEBs 
code uses the socket 'select' mechanism which I presume plays a role in 
this working. I do not understand why this same mechanism fails when we 
use SSL.


Thanks again
-Jim
I suppose re-reading what I wrote it does sound like I am pointing a 
finger but that was not my intention. I'm looking for advice on how to 
fix our problem. I was hoping for a "silver bullet" - just call this 
function and all will be fine type of answer. Wishful thinking, I know.


Blocking socket operations are very difficult to use. I strongly urge 
you to

switch to non-blocking operations.
Having only done minimal socket programming, I'm in a bit of a steep 
learning curve right now. Other then understanding what a blocking and 
non-blocking operation is, I don't fully understand the ramifications of 
switching to non-blocking I/O. Compounding this issue is the third party 
code, which was clearly written with blocking I/O :(


I appreciate your time!

Thanks
-Jim

__
OpenSSL Project http://www.openssl.org
User Support Mailing List
[EMAIL PROTECTED]
Automated List Manager   
[EMAIL PROTECTED]





__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Marshall

David Schwartz wrote:

We are working on a threaded solution but right now we are using some
third-party code (Webs 2.18) which is single threaded.


That's fine, but if you using blocking calls in a single-threaded
application, you can really only handle one client at a time.
I know :( Which is why we are writing our own code now, but that doesn't 
fix the current problem.



I just let telnet connect and don't type anything. The connection
remains 'ESTABLISHED'. and no other client connections will work (e.g.
wclient2). This is basically the behavior we are seeing with our
application. A client connects and never sends any data, so it remains
connected, hence blocking any other client from communicating.


The application elected to block until the negotiation was finished. That's
a pretty dumb thing for an application to do.


This seems to me (in my very novice opinion) to be an issue with
openSSL, in the sense that if it doesn't get a handshake (or some other
data) on the connection it should close the connection after a period of
time.


OpenSSL tries to make SSL connections act like regular TCP connections. This
is exactly what TCP does. So your application would have this exact same
problem with or without OpenSSL. As such, how can you blame it on OpenSSL?
I suppose re-reading what I wrote it does sound like I am pointing a 
finger but that was not my intention. I'm looking for advice on how to 
fix our problem. I was hoping for a "silver bullet" - just call this 
function and all will be fine type of answer. Wishful thinking, I know.


Blocking socket operations are very difficult to use. I strongly urge you to
switch to non-blocking operations.
Having only done minimal socket programming, I'm in a bit of a steep 
learning curve right now. Other then understanding what a blocking and 
non-blocking operation is, I don't fully understand the ramifications of 
switching to non-blocking I/O. Compounding this issue is the third party 
code, which was clearly written with blocking I/O :(


I appreciate your time!

Thanks
-Jim

__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


RE: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread David Schwartz

> We are working on a threaded solution but right now we are using some
> third-party code (Webs 2.18) which is single threaded.

That's fine, but if you using blocking calls in a single-threaded
application, you can really only handle one client at a time.

> I'm also not sure I understand your answer "This is the way TCP works".
> When we disconnect the network cable the connection never times out (we
> left it for at least 30 minutes).

Right.

> Let me give you (and others) a more concrete example. I took the
> wserver2 sample from http://www.rtfm.com/openssl-examples/ and built on
> FC 6. I started the server like so
>
> ./wserver2 -n
>
> Then from another machine I telnet to the wserver2 machine
>
> telnet  4433
>
> I just let telnet connect and don't type anything. The connection
> remains 'ESTABLISHED'. and no other client connections will work (e.g.
> wclient2). This is basically the behavior we are seeing with our
> application. A client connects and never sends any data, so it remains
> connected, hence blocking any other client from communicating.

The application elected to block until the negotiation was finished. That's
a pretty dumb thing for an application to do.

> This seems to me (in my very novice opinion) to be an issue with
> openSSL, in the sense that if it doesn't get a handshake (or some other
> data) on the connection it should close the connection after a period of
> time.

OpenSSL tries to make SSL connections act like regular TCP connections. This
is exactly what TCP does. So your application would have this exact same
problem with or without OpenSSL. As such, how can you blame it on OpenSSL?

Blocking socket operations are very difficult to use. I strongly urge you to
switch to non-blocking operations.

> I agree it requires a time-out, but since we are in the openSSL library
> (waiting for the hello message) there is nothing I can do in my
> application. Is there a setting I can pass to SSL to tell it to time-out?

Just don't make the call that's giving you problems. You are calling a
function and asking it to block until the negotiation is completed or it is
clear it cannot be. Non-blocking operations are available, so use them.

> As for 'keepalive', I don't think that will help if the telnet session
> is still connected (e.g. telnet will ack the keep-alive requests).

If we're talking about not just a disconnected network cable but the whole
universe of possible bad things a client can do, yeah. You need timeouts
around everything you do. That's an advantage of non-blocking operations,
they are easy to time out.

DS

__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Fox




Date: Thu, 19 Jul 2007 16:09:59 -0400
From: Jim Marshall <[EMAIL PROTECTED]>
To: openssl-users@openssl.org
Reply-To: openssl-users@openssl.org
Subject: Re: How to prevent SSL from blocking from Network interruption

Victor Duchovni wrote:

On Thu, Jul 19, 2007 at 03:54:55PM -0400, Jim Marshall wrote:


I'm also not sure I understand your answer "This is the way TCP works". 
When we disconnect the network cable the connection never times out (we 
left it for at least 30 minutes).


TCP only disconnects quickly when a writer re-tranmits unacked data.
TCP never[1] disconnects when a reader waits for more data, this requires
an application timeout.
I agree it requires a time-out, but since we are in the openSSL library 
(waiting for the hello message) there is nothing I can do in my application. 
Is there a setting I can pass to SSL to tell it to time-out?


As for 'keepalive', I don't think that will help if the telnet session is 
still connected (e.g. telnet will ack the keep-alive requests).




That's why you use non-blocking rather than keepalive.

Jim
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Marshall

Victor Duchovni wrote:

On Thu, Jul 19, 2007 at 03:54:55PM -0400, Jim Marshall wrote:


I'm also not sure I understand your answer "This is the way TCP works". 
When we disconnect the network cable the connection never times out (we 
left it for at least 30 minutes).


TCP only disconnects quickly when a writer re-tranmits unacked data.
TCP never[1] disconnects when a reader waits for more data, this requires
an application timeout.
I agree it requires a time-out, but since we are in the openSSL library 
(waiting for the hello message) there is nothing I can do in my 
application. Is there a setting I can pass to SSL to tell it to time-out?


As for 'keepalive', I don't think that will help if the telnet session 
is still connected (e.g. telnet will ack the keep-alive requests).


-Jim

__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Sansing
Once a TCP session is in the Established state, it stays that way
until a FIN or RST is received from the opposite host, or if a
packet times out.  This is the purpose of keepalive.  However,
the default time for keepalive is a system wide setting and is
usually very long, and may not be appropriate for your application.
In my application, I have implemented my own keepalive that
is sent every 15 sec. if there is no data sent/recv'd in that time.

However, I have found that it can be difficult to determine the
status of the session, even with this feature.  Therefore, I also
check the socket option, SO_ERROR, before and after sending
or receiving data.  Until I implemented all of this, my app did
not consistently drop sessions when the other side just went away.
I have tested this implementation successfully both with and
without SSL.

Later . . .   Jim

Jim Marshall wrote:
> Hi Jim - thanks for the reply. See comments in-line
> Jim Fox wrote:
>>
>>>
>>> I have a single threaded test application (Red Hat Linux release 9 -
>>> Shrike), OpenSSL 0.9.8.  I found that it's possible to permanently
>>> hang a thread receiving SSL calls if a network interruption occurs
>>> during an established connection.
>>>
>>
>> This is the way TCP works.  There's a couple of minute timeout
>> built into it.   You can circumvent this behavior by setting
>> your ssl sockets to non-blocking, but if you can switch to
>> threaded processing that'd be the easier way to go.
> We are working on a threaded solution but right now we are using some
> third-party code (Webs 2.18) which is single threaded.
>
> I'm also not sure I understand your answer "This is the way TCP
> works". When we disconnect the network cable the connection never
> times out (we left it for at least 30 minutes).
>
> Let me give you (and others) a more concrete example. I took the
> wserver2 sample from http://www.rtfm.com/openssl-examples/ and built
> on FC 6. I started the server like so
>
> ./wserver2 -n
>
> Then from another machine I telnet to the wserver2 machine
>
> telnet  4433
>
> I just let telnet connect and don't type anything. The connection
> remains 'ESTABLISHED'. and no other client connections will work (e.g.
> wclient2). This is basically the behavior we are seeing with our
> application. A client connects and never sends any data, so it remains
> connected, hence blocking any other client from communicating.
>
> This seems to me (in my very novice opinion) to be an issue with
> openSSL, in the sense that if it doesn't get a handshake (or some
> other data) on the connection it should close the connection after a
> period of time.
>
> Thanks
> Jim
>
> __
> OpenSSL Project http://www.openssl.org
> User Support Mailing Listopenssl-users@openssl.org
> Automated List Manager   [EMAIL PROTECTED]
>
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: BF-CBC

2007-07-19 Thread Marek Marcola
Hello,
> i find an information about add new chipers BF-CBC but find nothing.
> may be somone ever find thid?
Look at:
http://www.openssl.org/docs/crypto/blowfish.html#
for low-level API, or use:
EVP_CIPHER_CTX_init(&ctx);
EVP_EncryptInit_ex(&ctx, EVP_bf_cbc(), NULL, key, iv);
EVP_EncryptUpdate(&ctx, outbuf, &outlen, inbuf, n);
EVP_EncryptFinal_ex(&ctx, outbuf, &outlen);
EVP_DecryptUpdate(&ctx, outbuf, &outlen, inbuf, n)
EVP_CIPHER_CTX_cleanup(&ctx);
for high level envelope API.

Best regards,
-- 
Marek Marcola <[EMAIL PROTECTED]>

__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Victor Duchovni
On Thu, Jul 19, 2007 at 03:54:55PM -0400, Jim Marshall wrote:

> Hi Jim - thanks for the reply. See comments in-line
> Jim Fox wrote:
> >
> >>
> >>I have a single threaded test application (Red Hat Linux release 9 - 
> >>Shrike), OpenSSL 0.9.8.  I found that it's possible to permanently 
> >>hang a thread receiving SSL calls if a network interruption occurs 
> >>during an established connection.
> >>
> >
> >This is the way TCP works.  There's a couple of minute timeout
> >built into it.   You can circumvent this behavior by setting
> >your ssl sockets to non-blocking, but if you can switch to
> >threaded processing that'd be the easier way to go.
> We are working on a threaded solution but right now we are using some 
> third-party code (Webs 2.18) which is single threaded.
> 
> I'm also not sure I understand your answer "This is the way TCP works". 
> When we disconnect the network cable the connection never times out (we 
> left it for at least 30 minutes).

TCP only disconnects quickly when a writer re-tranmits unacked data.
TCP never[1] disconnects when a reader waits for more data, this requires
an application timeout.

-- 
Viktor.

[1] When you enable TCP keep-alives, the reader sends periodic zero
length probes and so becomes a writer, thereby detecting a broken
connection. The default keepalive timer from BSD 4.2 days is 2 hours,
but this is nowdays tunable using sysctl, ndd, ...
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Marshall

Hi Jim - thanks for the reply. See comments in-line
Jim Fox wrote:




I have a single threaded test application (Red Hat Linux release 9 - 
Shrike), OpenSSL 0.9.8.  I found that it's possible to permanently 
hang a thread receiving SSL calls if a network interruption occurs 
during an established connection.




This is the way TCP works.  There's a couple of minute timeout
built into it.   You can circumvent this behavior by setting
your ssl sockets to non-blocking, but if you can switch to
threaded processing that'd be the easier way to go.
We are working on a threaded solution but right now we are using some 
third-party code (Webs 2.18) which is single threaded.


I'm also not sure I understand your answer "This is the way TCP works". 
When we disconnect the network cable the connection never times out (we 
left it for at least 30 minutes).


Let me give you (and others) a more concrete example. I took the 
wserver2 sample from http://www.rtfm.com/openssl-examples/ and built on 
FC 6. I started the server like so


./wserver2 -n

Then from another machine I telnet to the wserver2 machine

telnet  4433

I just let telnet connect and don't type anything. The connection 
remains 'ESTABLISHED'. and no other client connections will work (e.g. 
wclient2). This is basically the behavior we are seeing with our 
application. A client connects and never sends any data, so it remains 
connected, hence blocking any other client from communicating.


This seems to me (in my very novice opinion) to be an issue with 
openSSL, in the sense that if it doesn't get a handshake (or some other 
data) on the connection it should close the connection after a period of 
time.


Thanks
Jim

__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


Re: How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Fox




I have a single threaded test application (Red Hat Linux release 9 - Shrike), 
OpenSSL 0.9.8.  I found that it's possible to permanently hang a thread 
receiving SSL calls if a network interruption occurs during an established 
connection.




This is the way TCP works.  There's a couple of minute timeout
built into it.   You can circumvent this behavior by setting
your ssl sockets to non-blocking, but if you can switch to
threaded processing that'd be the easier way to go.


Sometime after you create each socket do something like this:

  int sl = 1;
  SSL *ssl;
  ...
  // set ssl socket nonblocking
  if (BIO_socket_ioctl(SSL_get_fd(ssl),FIONBIO,&sl) < 0) {
// report an error
  }


Then you have to deal with the likelyhood that every 
ssl operation will return a "would block": either

an SSL_ERROR_WANT_READ or an SSL_ERROR_WANT_WRITE.

When you get either of these you can retry the same operation
later.


Here's a library that demonstrates non-blocking SSL IO:

  http://staff.washington.edu/fox/ezs/


Jim







__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


How to prevent SSL from blocking from Network interruption

2007-07-19 Thread Jim Marshall

Good Evening folks:

If someone could push (or kick) me in the right direction, that would be 
much appreciated.


I have a single threaded test application (Red Hat Linux release 9 - 
Shrike), OpenSSL 0.9.8.  I found that it's possible to permanently hang 
a thread receiving SSL calls if a network interruption occurs during an 
established connection.


A simple way to reproduce it would be to
1.  Telnet to the SSL port on another machine, don't actually enter any 
data (triggers a client/server Syn and Ack TCP exchnage, but no data 
PUSHes).


2.  netstat shows the connection is now ESTABLISHED on the telnet 
machine and the SSL server machine.


3.  Simulate an outage by taking out the network jack on the client 
machine. telnet sees this and closes it's connection.


4.  The SSL Server thread show the connection as ESTABLISHED (no TCP 
signal is sent otherwise) and waits for data that never comes.  Stack 
trace looks something like this:


(gdb) bt
#0  0xe410 in __kernel_vsyscall ()
#1  0xf7e711fb in __read_nocancel () from /lib/libpthread.so.0
#2  0xf7af3eec in BIO_sock_non_fatal_error ()
   from /usr/ws/server/cserver/../../lib/linux/libcrypto.so.0.9.8
#3  0x000a in ?? ()

I built openSSL 0.9.8 with the default configure options.

How could this operation be timed out?

Is this a problem with the way I am opening SSL ports?
Or the TCP layer underneath?
Any API suggestions for opening the SSL port?

Thanks again for any replies,
Jim

__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   [EMAIL PROTECTED]


OpenSSL for Visual Studio.

2007-07-19 Thread Abhishek Tripathi

Hi Friends,

Can any body help me out to get the MS Visual Studio Workspace of
OpenSSL ?

Regards,
Abhishek