RE: SSL overhead
Hi, Just a shy question: are you sure you can't do what you want by just using BIO_new_bio_pair, connect one to SSL and deal with the other one for networking, like: BIO_new_bio_pair(pInternalBio, BUF_SIZE, pNetworkBio, BUF_SIZE); SSL_set_bio(pSSL, pInternalBio, pInternalBio); For BUF_SIZE you can choose anything you like, but if it's too small you'll have to issue more network calls (and thus go into the kernel more often). I use 16384 because it's the max. TLS record size. Anyway, man BIO_new_bio_pair has a neat picture explaining this approach. I am using it to work over a custom network layer that works on Windows and most flavours of UNIX. On Windows we use IOCP, and on UNIX we use poll(). Everything works flawlessly - OpenSSL is completely detached from the networking code, really just a pipe in some respects. One issue is, that you need another buffer underneath, *unless* you use the non-copying functions like BIO_nread0/BIO_nread, BIO_nwrite0/BIO_nwrite, which unfortunately are not officially supported/documented. Cheers, Michael. -Original Message- From: Tomas Neme [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 15, 2008 4:24 PM To: openssl-users@openssl.org Subject: Re: SSL overhead OK, I think I'll implement my own BIO. My C is good, although not brilliant, but I can do it. What stopped me from trying that approach in the first place was that the BIO* functions are so many and that I don't know which ones do I actually *need* to reimplement (that is, which ones is the engine going to use), but I'll dig into it. Thanks -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
* Tomas Neme wrote on Wed, Apr 16, 2008 at 17:44 -0300: The problem is that SSL_pending might return 0 if there's stuff on the socket, but isn't enough to decode a full record. If you want SSL to be transparent, why are you interested in intermediate protocol-specific data which would not be returned by SSL read anyway? (isn't calling SSL_pending in the loop and if so, calling SSL_read() to get the data what you need to do?) oki, Steffen About Ingenico Throughout the world businesses rely on Ingenico for secure and expedient electronic transaction acceptance. Ingenico products leverage proven technology, established standards and unparalleled ergonomics to provide optimal reliability, versatility and usability. This comprehensive range of products is complemented by a global array of services and partnerships, enabling businesses in a number of vertical sectors to accept transactions anywhere their business takes them. www.ingenico.com This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
On Wed, Apr 16, 2008, Tomas Neme wrote: does one of the flags in the BIO object say anything about the blocking nature of the BIO? For example, does SSL_connect return automatically or not depending on the BIO flags, or on the BIO_write and read return values? If a BIO is blocking it should not return until some data has been processed or an error occurs. If it is non-blocking if data can't be processed immediately (usually an attempt to read and no data is available) the BIO returns -1 and the retry flag is set for the appropriate operation. See the macros BIO_set_retry_*() in bio.h Steve. -- Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage OpenSSL project core developer and freelance consultant. Homepage: http://www.drh-consultancy.demon.co.uk __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
My first stumbles with OpenSSL [was: SSL overhead]
All who have answered: Thanks a lot.
Yes, I understood that the poll() should read to a buffer last night,
in the shower. You know how that works ;). But I also understood that
I had misunderstood the way the poll and read methods were being
used, and read will be called no matter what poll returns, until
all the data has been received, or an error is returned, so it's fine,
Now, I've finished the core coding of the thing up to the point where
I can actually test things.. and it doesn't work. SSL_Connect doesn't
succeed, and my read and write functions are never called. Here's some
sample code, if anyone cares for checking what might I be missing...
I'm sure it's a quite stupid mistake, but I just can't figure it out,
never having coded OSSL apps before.
There's quite a lot of non-standard methods, but their working is
pretty straight forward. All I'm doing is creating an SSL_Client
instance and trying to a nssl listening in localhost:, the socket
gets connected, but the prints in BIO_larvita_write and _read are
never done, and I'm guessing those are used for the handshake.
SSL_Connect returns with -2 and the
printf(%s\n,ERR_error_string(err,NULL)); spits this:
error:0001:lib(0):func(0):reason(1).
static BIO_METHOD larvita_method = {
BIO_TYPE_LARVITA,
larvita socket,
larvita_write,
larvita_read,
NULL,
NULL,
NULL,
larvita_new,
larvita_free,
NULL,
};
static int larvita_write(BIO* bio, const char* buf, int len) {
printf(larvita_write(%s)\n,buf);
int ret;
//clear_sys_error();
ret = ((TCP_Client*)bio-ptr)-send_data((Uint8*)buf,len);
BIO_clear_retry_flags(bio);
if(ret == 0) {
BIO_set_retry_write(bio);
}
return ret;
}
static int larvita_read(BIO* b, char* out, int len) {
printf(larvita_read()\n);
int ret = 0;
BIO_clear_retry_flags(b);
ret = ((TCP_Client*)b-ptr)-read((Uint8*)out,len);
if(ret == 0) {
BIO_set_retry_read(b);
}
return ret;
}
static int larvita_new(BIO* bio) {
bio-init = 0;
bio-num = 0;
bio-ptr = NULL;
bio-flags = 0;
return 1;
}
static int larvita_free(BIO* bio) {
if(bio == NULL) return 0;
if(bio-shutdown) {
if(bio-init) {
if(bio-ptr) {
memdelete((TCP_Client*)bio-ptr);
bio-ptr = NULL;
}
}
bio-init = 0;
bio-flags = 0;
}
return 1;
}
BIO_METHOD *BIO_s_larvita_client(){
return (larvita_method);
}
BIO *BIO_new_larvita_client(TCP_Client* client){
BIO* ret = NULL;
if((ret = BIO_new(BIO_s_larvita_client( {
ret-ptr = client;
}
return ret;
}
Error SSL_Client::connect_to_host(String p_host,int p_port) {
printf(ssl connecting to host %s:%d\n,p_host.ascii().get_data(),p_port);
Error ret = this-client-connect_to_host(p_host,p_port);
printf(connected? err: %d\n,ret);
if(ret == OK) {
int err;
while((err = SSL_connect(this-ssl)) 0) {
printf(err = %d\n,err);
switch((err = SSL_get_error(this-ssl,err))) {
case SSL_ERROR_WANT_READ:
break;
case SSL_ERROR_WANT_WRITE:
break;
default:
printf(%s\n,ERR_error_string(err,NULL));
break;
}
}
printf(ssl opened? err: %d\n,err);
}
return ret;
}
SSL_CTX* SSL_Client::ctx = NULL;
void SSL_Client::initialize_ctx() {
SSL_library_init();
SSL_load_error_strings();
ERR_load_crypto_strings();
SSL_Client::ctx = SSL_CTX_new(SSLv23_method());
SSL_CTX_set_cipher_list(SSL_Client::ctx,CIPHERS_LIST);
}
SSL_Client::SSL_Client() {
if(!SSL_Client::ctx)
SSL_Client::initialize_ctx();
this-client = TCP_Client::create();
this-ssl = SSL_new(this-ctx);
this-bio = BIO_new_larvita_client(this-client);
SSL_set_bio(this-ssl,this-bio,this-bio);
SSL_set_connect_state(this-ssl);
}
--
|_|0|_|
|_|_|0|
|0|0|0|
(\__/)
(='.'=)This is Bunny. Copy and paste bunny
()_() to help him gain world domination.
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
One more question: I'm working on an ansynchronous framework, and there's a poll method that gets called in each iteration. In our sockets, this method's supposed to do whatever needs doing, and return how many bytes are waiting to be read from the socket, so the return value should be the value of SSL_pending, if I want SSL to be transparent. The problem is that SSL_pending might return 0 if there's stuff on the socket, but isn't enough to decode a full record. And as long as poll() returns 0, no calls to read() which triggers a call to SSL_read() will be made, so I need to call SSL_read() in the calls to poll(), or to find some other way to force SSL to process a little bit of data. Should I call SSL_read() with a num value of 0? will that work? or is there something else I'm missing here? Thanks Tomas -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
does one of the flags in the BIO object say anything about the blocking nature of the BIO? For example, does SSL_connect return automatically or not depending on the BIO flags, or on the BIO_write and read return values? T -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: SSL overhead
One more question: I'm working on an ansynchronous framework, and there's a poll method that gets called in each iteration. In our sockets, this method's supposed to do whatever needs doing, and return how many bytes are waiting to be read from the socket, so the return value should be the value of SSL_pending, if I want SSL to be transparent. The problem is that SSL_pending might return 0 if there's stuff on the socket, but isn't enough to decode a full record. And as long as poll() returns 0, no calls to read() which triggers a call to SSL_read() will be made, so I need to call SSL_read() in the calls to poll(), or to find some other way to force SSL to process a little bit of data. Should I call SSL_read() with a num value of 0? will that work? or is there something else I'm missing here? This seems like a strange and inefficient architecture. Why receive the data twice, once in a some kind of 'poll' operation and then again in a 'read' operation. Either the 'poll' operation should just say do something or the 'poll' operation should provide the data. Why not just call SSL_read in the 'poll' operation and if the return value is non-zero, stash the data in a buffer and return the number of bytes you read? That way you won't need to call into the SSL engine again to get the data you knew you needed anyway. If the purpose of the 'poll' operation is just to say yes, there's something to do, then you should not be calling the 'poll' operation unless directed to by the SSL framework. If the SSL framework returns 'WANT_READ', then you know that no further progress will be possible until either you can read more data from the socket or something else happens. In that case, your 'poll' can check the real socket. Otherwise, it is a mistake to wait since the data may already be there. In sum: 1) Check the socket for readability only if the SSL engine last gave you a WANT_READ and you've made no further progress since then. Otherwise, just call SSL_read. 2) Don't try to figure out how many bytes there are, just get them, and then not only will you know how many there are, but you'll know what they are too. You'll definitely need that later, so don't force the SSL engine to do the work of processing them twice. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
OK, I think I'll implement my own BIO. My C is good, although not brilliant, but I can do it. What stopped me from trying that approach in the first place was that the BIO* functions are so many and that I don't know which ones do I actually *need* to reimplement (that is, which ones is the engine going to use), but I'll dig into it. Thanks -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
Is there any reference of what do the different members of the BIO
objects mean? method, ptr, next_bio and prev_bio are pretty obvious,
but the rest, I don't know, and I haven't been able to find any
details out there.. I'll keep googling, but if anyone knows of a good
detail out there, please point me
Thanks
Tomas
struct bio_st
{
BIO_METHOD *method;
/* bio, mode, argp, argi, argl, ret */
long (*callback)(struct bio_st *,int,const char *,int, long,long);
char *cb_arg; /* first argument for the callback */
int init;
int shutdown;
int flags; /* extra storage */
int retry_reason;
int num;
void *ptr;
struct bio_st *next_bio;/* used by filter BIOs */
struct bio_st *prev_bio;/* used by filter BIOs */
int references;
unsigned long num_read;
unsigned long num_write;
CRYPTO_EX_DATA ex_data;
};
--
|_|0|_|
|_|_|0|
|0|0|0|
(\__/)
(='.'=)This is Bunny. Copy and paste bunny
()_() to help him gain world domination.
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
On Tue, Apr 15, 2008, Tomas Neme wrote: Is there any reference of what do the different members of the BIO objects mean? method, ptr, next_bio and prev_bio are pretty obvious, but the rest, I don't know, and I haven't been able to find any details out there.. I'll keep googling, but if anyone knows of a good detail out there, please point me Most of them you don't need to worry about. I'd suggest looking at the fd BIO and copy some of the functionality from that. If you need non blocking I/O I'd suggest getting blocking going first which is just a case of supplying custom read/write routines in the BIO_METHOD. Once that is working you can get sort out the retry mechanism. Steve. -- Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage OpenSSL project core developer and freelance consultant. Homepage: http://www.drh-consultancy.demon.co.uk __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
Most of them you don't need to worry about. I'd suggest looking at the fd BIO and copy some of the functionality from that. OK, I will. I just wondered because mem_new (the new method of BIO_s_mem) sets shutdown = 1 ; init = 1; num = -1 and rtcp does init = 1 ; num = 0 ; flags = 0 ; so I'd like to know what's the difference, and what do I set to what to achieve which effect. If you need non blocking I/O I'd suggest getting blocking going first which is just a case of supplying custom read/write routines in the BIO_METHOD. Once that is working you can get sort out the retry mechanism. I'll think about it. In our library most blocking methods are just calls to the non-blocking inside a while-loop, but that's my partner's style. Thanks a lot Tomás -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
Stephen: Most of them you don't need to worry about. I'd suggest looking at the fd BIO and copy some of the functionality from that. I also need to know which of the BIO_CTRL_* params do I have to implement in my _ctrl function. Is there a reason why you said to look at the fd BIO instead of the conn BIO or socket BIO? It would seem that the socket BIO handles the socket abstraction, but it does not handle calls to BIO_S_CONNECT. Actually, a grep for BIO_S_CONNECT returns only bss_conn.. why is this? shouldn't sockets have a call to connect too? Tomás -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
On Tue, Apr 15, 2008 at 04:22:03PM -0300, Tomas Neme wrote: Stephen: Most of them you don't need to worry about. I'd suggest looking at the fd BIO and copy some of the functionality from that. I also need to know which of the BIO_CTRL_* params do I have to implement in my _ctrl function. Is there a reason why you said to look at the fd BIO instead of the conn BIO or socket BIO? If you use the biopair approach, you don't need any of this, you just handle requests to read/write data (with a bit of care to always flush the write buffers before attempting to read). -- Viktor. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
It would seem that the socket BIO handles the socket abstraction, but it does not handle calls to BIO_S_CONNECT. Actually, a grep for BIO_S_CONNECT returns only bss_conn.. why is this? shouldn't sockets have a call to connect too? sorry, I forgot that you create the BIO_socket with the socket already connected T -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
SSL overhead
We have our own TCP implementation, and we're thinking of using a BIO_s_mem to add an SSL layer to it. The plan is: read the socket, put the encrypted data into the ssl object's BIO, and then do a read from it. Likewise, produce the data, feed it into OpenSSL, and then take the data from the BIO and put it into the socket. We want to minimize memory allocation, working with a fixed-size buffer, so I'd like to know if there's a way to know the size overhead on SSL headers, so I know that if I feed it, say 200B blocks, I have to read with a 350B buffer, or something like that. Is there some way to know that? thanks Tomas -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
I'm getting started with ssl, and there's a lot of things I don't get about the library, small, and quite a lot of them Isn't there an IRC channel, or some kind of faster communication way than this? The documentation's poor at best, and I don't completely get the general concepts. From reading examples I figure that only the BIO_f_ssl does encryption-decryption when written into? so what should I do if I want to provide an api that has functions b_encrypt and encrypt_flush? I have to use OpenSSL with memory buffers, because I can't give it direct access to sockets, so I'm wondering how to handle the handshakes and whatnot in a non-blocking way. Do I need 2 BIO_f_ssls, one for imput and one for output, or what? I just want an object which will take data, encrypt it and put it in a buffer, which I can use to write into the socket, and that will read encrypted data that I will take from a socket, and put the decrypted version into a buffer, so I can pass it to my http client. I don't want the SSL object to handle the sockets directly because this has to be portable, but I don't fully understand which kinds of BIOs do I need to model this data flow, and how do I handle the handshakes and the such. Tomas --- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
general concepts. From reading examples I figure that only the BIO_f_ssl does encryption-decryption when written into? so what should Or doing SSL_write into a SSL object with a BIO_s_mem object as the write-bio will write the encrypted data into it? T -- |_|0|_| |_|_|0| |0|0|0| (\__/) (='.'=)This is Bunny. Copy and paste bunny ()_() to help him gain world domination. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: SSL overhead
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tomas Neme wrote: | The documentation's poor at best, and I don't completely get the | general concepts. From reading examples I figure that only the | BIO_f_ssl does encryption-decryption when written into? so what should | I do if I want to provide an api that has functions b_encrypt and | encrypt_flush? If you want to do SSL, you should stick with BIO_f_ssl. If you want to do some other encryption, you can use BIO_f_cipher. | I have to use OpenSSL with memory buffers, because I | can't give it direct access to sockets, so I'm wondering how to handle | the handshakes and whatnot in a non-blocking way. How good is your C ? you could look into bio_s_socket and implement your own bio. After you understood the concept behind BIO, it is simple to do... | Do I need 2 | BIO_f_ssls, one for imput and one for output, or what? No. you treat it almost as a normal socket: You write data to be encrypted (and sent to the peer) into it and read decrypted data (received from the peer) from it. | I don't want the SSL | object to handle the sockets directly because this has to be portable, | but I don't fully understand which kinds of BIOs do I need to model | this data flow, and how do I handle the handshakes and the such. The SSL bio is just a filter. It relies on an underlying BIO to do the low level IO communication. This way you can use different network implementations to transport the encrypted data. Goetz - -- DMCA: The greed of the few outweighs the freedom of the many -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIA8EP2iGqZUF3qPYRAr0EAJ0fVnuLUsKWjJiaNdjxuQrTx5XeAwCfa+jG dDf6VRyXthXB+QJ4ye6i/cQ= =Adx8 -END PGP SIGNATURE- __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: SSL overhead
The documentation's poor at best, and I don't completely get the general concepts. From reading examples I figure that only the BIO_f_ssl does encryption-decryption when written into? so what should I do if I want to provide an api that has functions b_encrypt and encrypt_flush? I think you have a wrong notion in your head that's leading you astray. You are thinking about an SSL connection as a module that you put some plaintext into, some encrypted data comes out, and then you send that on a socket, you get some encrypted data back, you feed that into the SSL connection, and you get some plaintext back. While that certainly *sometimes* happens, other times it doesn't happen. Sometimes you want to hand plaintext to the SSL engine, but it can't encrypt it yet because it hasn't negotiate the key. Sometimes data comes in from the socket that's protocol data and doesn't correspond to application data. Your whole concept of how big does my buffer have to be is based on the mistaken notion that it's useful to try to track what goes in to its corresponding output. It's not. That's the SSL engine's job. It is better to think the the SSL engine as a box that has four 'links'. One is the one you feed plaintext into so that it can be encrypted and sent. One is the one encrypted data comes out of. One is the one that you get decrypted plaintext from. One is the one that data that comes in from the wire is fed into. The relationship between these four links to 100% the SSL engine's business, and you shouldn't make any assumptions about it. If you hand plaintext to the SSL engine, at some point it might hand encrypted data to you to send over the Link. Or it might not. That's not your business. If you get encrypted data from the other end, give it to the SSL engine. It might output some plaintext for you to treat as received, or it might not. That's not your business. You have two general approaches to choose from. You can implement a BIO and OpenSSL will call into your BIO whenever it needs to send or receive data to or from the other end. This will result in your calls to SSL_read/SSL_write making calls into your own send/receive functions when the SSL engine needs to interact with the link to the other end. This is probably the simpler approach and it makes deadlocks easier to avoid. The other way is to use BIO pairs. With A BIO pair, you actually independently manage all four links independently. When you get data from the other end of the SSL link, you will actually 'write' it onto the appropriate link. When you make forward progress, you will actually call a 'read' function to get data from OpenSSL that you send to the other end. This is a bit trickier to do as it is easy to inadvertently cause deadlock, especially in non-blocking implementations. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
