Re: ASN1 non blocking I/O...
From: Dan Kegel [EMAIL PROTECTED] dank BIO's should have been a well-separated layer, not an integral dank part of OpenSSL. I'm not sure I understand that argument. BIO's *are* separate, in their own "module", if you wish to express it that way. They just happen to be used by OpenSSL as transport, which I see as a feature, since that makes it possible for the application programmer to make his own (I'd like to see you do your own FILE * or thingy controlled through file descriptors). I wonder, is it really BIO's that are your problem, or the fact that SSL itself (the handshake, most particularly) makes non-blocking I/O a bit tricky? Just trying to set the record straight... -- Richard Levitte \ Spannvägen 38, II \ [EMAIL PROTECTED] Chairman@Stacken \ S-168 35 BROMMA \ T: +46-8-26 52 47 Redakteur@Stacken \ SWEDEN \ or +46-709-50 36 10 Procurator Odiosus Ex Infernis-- [EMAIL PROTECTED] Member of the OpenSSL development team: http://www.openssl.org/ Software Engineer, Celo Communications: http://www.celocom.com/ Unsolicited commercial email is subject to an archival fee of $400. See http://www.stacken.kth.se/~levitte/mail/ for more info. __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: cvs commit: openssl/crypto/evp bio_enc.c evp.h
Ben Laurie wrote: [EMAIL PROTECTED] wrote: /* read in at offset 8, read the EVP_Cipher * documentation about why */ - i=BIO_read(b-next_bio,(ctx-buf[8]),ENC_BLOCK_SIZE); + i=BIO_read(b-next_bio,(ctx-buf[BUF_OFFSET]),ENC_BLOCK_SIZE); BTW, AFAICS, this is not documented at all - anyone know what its about? I think this is so it can encrypt the buffer in place rather than have separate plain text and encrypted buffers. The way EVP works you can't have the same input and output buffers but because it handles a block at a time its legal to write the encrypted date one block behind the plain text. And presumably vice versa for decryption. Steve. -- Dr Stephen N. Henson. http://www.drh-consultancy.demon.co.uk/ Personal Email: [EMAIL PROTECTED] Senior crypto engineer, Celo Communications: http://www.celocom.com/ Core developer of the OpenSSL project: http://www.openssl.org/ Business Email: [EMAIL PROTECTED] PGP key: via homepage. __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: ASN1 non blocking I/O...
Dan Kegel wrote: Dr S N Henson wrote: The revised OpenSSL ASN1 code will eventually have support for I/O based operations in addition to the current memory based operations which need to have the entire structure in memory (tricky if its a 1Gb structure on tape). Doing blocking I/O is fairly straight forward. Handling non blocking I/O is quite a bit harder because the ASN1 decoder or encoder must be able to save its internal state and restart where it left off. So the question: how many people would want non blocking I/O support? If no one or hardly anyone wants or needs it then there isn't a lot of point. However if there's considerably demand it would be worth looking into. My comments refer to ASN1 I/O only. OpenSSL doesn't currently have I/O based ASN1 code at all and I'm seeing if people have a specific need to non blocking ASN1 I/O or if the simpler blocking I/O will suffice. Can you give us an example of how ASN1 I/O would be used? I'm having trouble imagining, since I'm from a land very far from ASN1. Its main use would be with huge PKCS#7 structures. If you don't want or need them then most things will be unchanged: most of it will still use memory base I/O because its quicker. In particular, from what I know so far, the app is unable to have one thread execute the 'cheap' parts of OpenSSL, and another execute the expensive crypto parts, partly because OpenSSL uses BIOs internally. This gets to be a bit of a problem when you're pushing the limits, trying to handle thousands of sessions with a handful of threads. I'd say that has very little to do with BIOs and more to do with the fact that the public key API has no concept of "non blocking operations" and this has filtered down to the SSL API. It may be possible to actually do what you want through some of the SSL state callbacks but I haven't looked at that in much detail. Something like tracking the states and then moving the execution to another thread when the state involve expensive crypto operations then shifting it back when they have finished. Steve. -- Dr Stephen N. Henson. http://www.drh-consultancy.demon.co.uk/ Personal Email: [EMAIL PROTECTED] Senior crypto engineer, Celo Communications: http://www.celocom.com/ Core developer of the OpenSSL project: http://www.openssl.org/ Business Email: [EMAIL PROTECTED] PGP key: via homepage. __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Hmm... (discoveries about BIGNUM code)
Constification can lead you to very nice discoveries when you really
start digging. The BIGNUM code can be quite a treasure of surprises.
For example, I discovered that the second argument to bn_sqr_recursive()
can safely be const (I looked through the downstream call tree, and
found that for all functions that are given that same argument,
there's no reason for them to touch it, and a constification of them
proved me right). This means that in the part of BN_sqr() where a
bn_wexpand() is made to the size k*2, followed by a bn_sqr_recursive(),
it's perfectly safe to use a variant of bn_dup_expand() that doesn't
BN_dup() when no actual expansion is done. However, if expansion is
actually needed, one must of course remember to free the expanded
version of the BIGNUM in question.
One might wonder what the importance of this is, until you realise
that a realloc() most often has the same cost as malloc() and a free()
put together, especially when we you have growing things, like a
BIGNUM with a data area the doubles in size :-).
So, I ended up making a bn_dup_expand2() that works just like
bn_dup_expand(), except that it doesn't do a BN_dup(), and the part in
BN_sqr() that I'm fiddling with now looks like this:
if (al == j)
{
const BIGNUM *tmp_bn = NULL;
BIGNUM *free_bn = NULL;
if ((tmp_bn = bn_dup_expand2(a,k*2,free_bn))
== NULL) goto err;
if (bn_wexpand(tmp,k*2) == NULL) goto err;
bn_sqr_recursive(rr-d,tmp_bn-d,al,tmp-d);
if (free_bn)
BN_free(free_bn);
}
bn_dup_expand2() looks like this:
const BIGNUM *bn_dup_expand2(const BIGNUM *b, int words, BIGNUM **free_r);
The return value will either be the pointer to an expanded copy of b
if expansion was needed, or simply b. *free_r will be assigned the
pointer to the expanded copy of b if expansion occured, otherwise it
will be NULL.
I'd love if someone could check my thinking so I don't do any booboo
that I'll just regret later. In the mean time, I'm going to see if
it's possible to do something similar with BN_mul()...
--
Richard Levitte \ Spannvägen 38, II \ [EMAIL PROTECTED]
Chairman@Stacken \ S-168 35 BROMMA \ T: +46-8-26 52 47
Redakteur@Stacken \ SWEDEN \ or +46-709-50 36 10
Procurator Odiosus Ex Infernis-- [EMAIL PROTECTED]
Member of the OpenSSL development team: http://www.openssl.org/
Software Engineer, Celo Communications: http://www.celocom.com/
Unsolicited commercial email is subject to an archival fee of $400.
See http://www.stacken.kth.se/~levitte/mail/ for more info.
__
OpenSSL Project http://www.openssl.org
Development Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
What are the guarantees of the data part of a BIGNUM?
[ Assumption in the following text: BIGNUM *foo; ] I get a little confused by the code I see. It looks like some functions (for example BN_sqr()) assumes the data in foo-d, from foo-top+1 to foo-dmax (inclusive), is filled with 0, while others, like BN_mul(), do not assume that and fill in a lot of 0's. The two pieces of code that I'm thinking of are: From BN_sqr() before I started fiddling: if (bn_wexpand(a,k*2) == NULL) goto err; if (bn_wexpand(tmp,k*2) == NULL) goto err; bn_sqr_recursive(rr-d,a-d,al,tmp-d); From BN_mul() before I started fiddling: bn_wexpand(a,k); bn_wexpand(b,k); bn_wexpand(t,k*4); bn_wexpand(rr,k*4); for (i=a-top; ik; i++) a-d[i]=0; for (i=b-top; ik; i++) b-d[i]=0; bn_mul_part_recursive(rr-d,a-d,b-d,al-j,j,t-d); And I'm pretty sure that at least bn_sqr_recursive assumes the part of a-d starting at a-d[al] is 0-filled... So, what's the deal with this? What are the real guarantees? Or, if I'm missing something (by not readin enough code), please point me in the right direction, as this is my first real dive in the BIGNUM code :-). -- Richard Levitte \ Spannvägen 38, II \ [EMAIL PROTECTED] Chairman@Stacken \ S-168 35 BROMMA \ T: +46-8-26 52 47 Redakteur@Stacken \ SWEDEN \ or +46-709-50 36 10 Procurator Odiosus Ex Infernis-- [EMAIL PROTECTED] Member of the OpenSSL development team: http://www.openssl.org/ Software Engineer, Celo Communications: http://www.celocom.com/ Unsolicited commercial email is subject to an archival fee of $400. See http://www.stacken.kth.se/~levitte/mail/ for more info. __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: cvs commit: openssl/crypto/evp bio_enc.c evp.h
Dr S N Henson wrote: Ben Laurie wrote: [EMAIL PROTECTED] wrote: /* read in at offset 8, read the EVP_Cipher * documentation about why */ - i=BIO_read(b-next_bio,(ctx-buf[8]),ENC_BLOCK_SIZE); + i=BIO_read(b-next_bio,(ctx-buf[BUF_OFFSET]),ENC_BLOCK_SIZE); BTW, AFAICS, this is not documented at all - anyone know what its about? I think this is so it can encrypt the buffer in place rather than have separate plain text and encrypted buffers. The way EVP works you can't have the same input and output buffers but because it handles a block at a time its legal to write the encrypted date one block behind the plain text. And presumably vice versa for decryption. I wondered about that, but since a block can be up to 32 bytes (now, but it used to be 16 or even 24 [what was that?], I believe), the offset is wrong, right? Also, it seems to work fine despite being wrong! Should I change it to EVP_MAX_BLOCK_SIZE? Also, why are there an extra two bytes in the buffer? Is this just caution (I hate that kind of caution, I'd rather find out I was wrong about the buffer size than allow some extra "for luck")? Cheers, Ben. __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: cvs commit: openssl/crypto/evp bio_enc.c evp.h
Dr S N Henson wrote: [EMAIL PROTECTED] wrote: Index: evp.h === RCS file: /e/openssl/cvs/openssl/crypto/evp/evp.h,v retrieving revision 1.51 retrieving revision 1.52 diff -u -r1.51 -r1.52 --- evp.h 2000/10/14 20:09:22 1.51 +++ evp.h 2000/11/12 02:13:38 1.52 @@ -121,6 +121,7 @@ #define EVP_MAX_MD_SIZE (16+20) /* The SSLv3 md5+sha1 type */ #define EVP_MAX_KEY_LENGTH 32 #define EVP_MAX_IV_LENGTH16 +#define EVP_MAX_BLOCK_LENGTH 32 #define PKCS5_SALT_LEN 8 /* Default PKCS#5 iteration count */ @@ -396,7 +397,7 @@ unsigned char oiv[EVP_MAX_IV_LENGTH]; /* original iv */ unsigned char iv[EVP_MAX_IV_LENGTH]; /* working iv */ - unsigned char buf[EVP_MAX_IV_LENGTH]; /* saved partial block */ + unsigned char buf[EVP_MAX_BLOCK_LENGTH];/* saved partial block */ int num;/* used by cfb/ofb mode */ void *app_data; /* application stuff */ Eh? Isn't the IV length the same as the block length? Nope. Rijndael's biggest IV is 16 bytes, but the biggest block is 32 bytes. I haven't checked why, so if someone wants to enlighten me... Cheers, Ben. __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: ASN1 non blocking I/O...
Dan - I agree with you whole heartly and this fact set me back tremendously. When I looked at the code I was really disappointed that there appeared to be way too much being done in the bowels of the api - and stuff that frankly does not belong there. So the question is - how do we fix it? On Sun, Nov 12, 2000 at 12:06:00AM -0800, Dan Kegel wrote: Dr S N Henson wrote: The revised OpenSSL ASN1 code will eventually have support for I/O based operations in addition to the current memory based operations which need to have the entire structure in memory (tricky if its a 1Gb structure on tape). Doing blocking I/O is fairly straight forward. Handling non blocking I/O is quite a bit harder because the ASN1 decoder or encoder must be able to save its internal state and restart where it left off. So the question: how many people would want non blocking I/O support? If no one or hardly anyone wants or needs it then there isn't a lot of point. However if there's considerably demand it would be worth looking into. My comments refer to ASN1 I/O only. OpenSSL doesn't currently have I/O based ASN1 code at all and I'm seeing if people have a specific need to non blocking ASN1 I/O or if the simpler blocking I/O will suffice. Can you give us an example of how ASN1 I/O would be used? I'm having trouble imagining, since I'm from a land very far from ASN1. I will say, though, that when writing protocol modules, I prefer to keep them WAY far separated from I/O, and make them able to handle arbitrary chunks of data. In general I'd prefer an api that made it possible for me to do my own nonblocking I/O and pass the results to the api rather than an API that tries to do I/O for me. Which reminds me of BIO's. (Uh-oh, he got me started!) Yes, they are very handy, but they have no place in the core of an SSL API, because for apps that use nonblocking I/O, they make it very hard for the app to control what's going on. (Yes, I've seen the state demo.) In particular, from what I know so far, the app is unable to have one thread execute the 'cheap' parts of OpenSSL, and another execute the expensive crypto parts, partly because OpenSSL uses BIOs internally. This gets to be a bit of a problem when you're pushing the limits, trying to handle thousands of sessions with a handful of threads. BIO's should have been a well-separated layer, not an integral part of OpenSSL. Don't repeat the same error with any new design. - Dan __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: Hmm... (discoveries about BIGNUM code)
From: Richard Levitte - VMS Whacker [EMAIL PROTECTED]
Actually, I'm starting to wonder if I'm losing my sanity. I've stared
at the code for some time now, and it does look like the expansion of
a in the code below is really needed, from reading bn_sqr_recursive()
and the stuff it calls...
if (al == j)
{
if (bn_wexpand(a,k*2) == NULL) goto err;
if (bn_wexpand(tmp,k*2) == NULL) goto err;
bn_sqr_recursive(rr-d,tmp_bn-d,al,tmp-d);
The conclusion is that it should be possible to remove the expansion
of a completely in the code above.
Could someone double-check for me, to make sure I'm going insane?
--
Richard Levitte \ Spannvägen 38, II \ [EMAIL PROTECTED]
Chairman@Stacken \ S-168 35 BROMMA \ T: +46-8-26 52 47
Redakteur@Stacken \ SWEDEN \ or +46-709-50 36 10
Procurator Odiosus Ex Infernis-- [EMAIL PROTECTED]
Member of the OpenSSL development team: http://www.openssl.org/
Software Engineer, Celo Communications: http://www.celocom.com/
Unsolicited commercial email is subject to an archival fee of $400.
See http://www.stacken.kth.se/~levitte/mail/ for more info.
__
OpenSSL Project http://www.openssl.org
Development Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
[STATUS] OpenSSL (Sun 12-Nov-2000)
OpenSSL STATUS Last modified at __ $Date: 2000/10/26 21:07:27 $ DEVELOPMENT STATE o OpenSSL 0.9.7: Under development... o OpenSSL 0.9.6: Released on September 24th, 2000 o OpenSSL 0.9.5a: Released on April 1st, 2000 o OpenSSL 0.9.5: Released on February 28th, 2000 o OpenSSL 0.9.4: Released on August09th, 1999 o OpenSSL 0.9.3a: Released on May 29th, 1999 o OpenSSL 0.9.3: Released on May 25th, 1999 o OpenSSL 0.9.2b: Released on March 22th, 1999 o OpenSSL 0.9.1c: Released on December 23th, 1998 RELEASE SHOWSTOPPERS AVAILABLE PATCHES o CA.pl patch (Damien Miller) IN PROGRESS o Steve is currently working on (in no particular order): ASN1 code redesign, butchery, replacement. EVP cipher enhancement. /* Proper (or at least usable) certificate chain verification. */ Private key, certificate and CRL API and implementation. Developing and bugfixing PKCS#7 (S/MIME code). Various X509 issues: character sets, certificate request extensions. o Geoff and Richard are currently working on: ENGINE (the new code that gives hardware support among others). o Richard is currently working on: UTIL (a new set of library functions to support some higher level functionality that is currently missing). Dynamic thread-lock support. Shared library support for VMS. NEEDS PATCH o non-blocking socket on AIX o $(PERL) in */Makefile.ssl o "Sign the certificate?" - "n" creates empty certificate file OPEN ISSUES o The Makefile hierarchy and build mechanism is still not a round thing: 1. The config vs. Configure scripts It's the same nasty situation as for Apache with APACI vs. src/Configure. It confuses. Suggestion: Merge Configure and config into a single configure script with a Autoconf style interface ;-) and remove Configure and config. Or even let us use GNU Autoconf itself. Then we can avoid a lot of those platform checks which are currently in Configure. o Support for Shared Libraries has to be added at least for the major Unix platforms. The details we can rip from the stuff Ralf has done for the Apache src/Configure script. Ben wants the solution to be really simple. Status: Ralf will look how we can easily incorporate the compiler PIC and linker DSO flags from Apache into the OpenSSL Configure script. Ulf: +1 for using GNU autoconf and libtool (but not automake, which apparently is not flexible enough to generate libcrypto) o The perl/ stuff needs a major overhaul. Currently it's totally obsolete. Either we clean it up and enhance it to be up-to-date with the C code or we also could replace it with the really nice Net::SSLeay package we can find under http://www.neuronio.pt/SSLeay.pm.html. Ralf uses this package for a longer time and it works fine and is a nice Perl module. Best would be to convince the author to work for the OpenSSL project and create a Net::OpenSSL or Crypt::OpenSSL package out of it and maintains it for us. Status: Ralf thinks we should both contact the author of Net::SSLeay and look how much effort it is to bring Eric's perl/ stuff up to date. Paul +1 WISHES o __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: cvs commit: openssl/crypto/evp bio_enc.c evp.h
Ben Laurie wrote: Dr S N Henson wrote: Eh? Isn't the IV length the same as the block length? Nope. Rijndael's biggest IV is 16 bytes, but the biggest block is 32 bytes. I haven't checked why, so if someone wants to enlighten me... Well they can enlighten me too then. I can't see how you can possibly use 32 byte blocks in CBC or CFB mode with a 16 byte IV. Steve. -- Dr Stephen N. Henson. http://www.drh-consultancy.demon.co.uk/ Personal Email: [EMAIL PROTECTED] Senior crypto engineer, Celo Communications: http://www.celocom.com/ Core developer of the OpenSSL project: http://www.openssl.org/ Business Email: [EMAIL PROTECTED] PGP key: via homepage. __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: cvs commit: openssl/crypto/evp bio_enc.c evp.h
Eh? Isn't the IV length the same as the block length? Nope. Rijndael's biggest IV is 16 bytes, but the biggest block is 32 bytes. I haven't checked why, so if someone wants to enlighten me... I too was under the assumption that the IV in use needed to be the same size as the block length. Since Rijndael encrypts at 128, 192, and 256 bits, the IV should be 16, 24, or 32 bytes in size. Not being a cryptography expert myself I'm afraid I provide little help to Ben's request. However, I DID notice that in the code downloaded from NIST that in the header file they specify a MAX_SIZE of 32 for the IV for cipher structure definition. However, the code in use is hard-coded 16 byte memcpy and pointer operations. Thus it appears that, no matter the size of block or key, the IV remains at 16 bytes for 128, 192, 256 bits keys. I think this has much to do with the following statement taken directly from the AES proposal, and I quote: 4. Specification Rijndael is an iterated block cipher with a variable block length and a variable key length. The block length and the key length can be independently specified to 128, 192 or 256 bits. Note: this section is intended to explain the cipher structure and not as an implementation guideline. For implementation aspects, we refer to Section 5. 4.1 The State, the Cipher Key and the number of rounds The different transformations operate on the intermediate result, called the State: Definition: the intermediate cipher result is called the State. The State can be pictured as a rectangular array of bytes. This array has four rows, the number of columns is denoted by Nb and is equal to the block length divided by 32. The Cipher Key is similarly pictured as a rectangular array with four rows. The number of columns of the Cipher Key is denoted by Nk and is equal to the key length divided by 32. I'd be curious to know what would happen to the IV code if the key size was increased beyond 256 bits as the proposal implies... For anyone wishing to see the code and spec for this cipher use the following link: http://csrc.nist.gov/encryption/aes/ (as if you guys don't know where it is! ;-) Hope this helps! __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
