On Sun, 20 Aug 2000, Dan Kegel wrote:
> [EMAIL PROTECTED] wrote:
> > I agree with you 100% adn I'm in the same boat. I suggest that if we
> > can't get people on board that we just go our own way.
>
> Whatever we come up with, I'm sure the classic OpenSSL API could
> be layered on top of it. Perhaps we could consider this effort an
> experimental refactoring of the OpenSSL codebase to improve its
> quality and reusability.
I agree with this approach. The threaded model is fine - and a very good
idea for many situations. Also the idea of just opening the socket in ssl
mode is the best way to handle client side communications. Lord knows as
a programmer I would not want to have to deal with the complexity of
non-blocking IO - it is simply overkill since a client only has to handle
a few connections at a time.
If the underlying code is properly constructed, the socket connect can run
on top of it easily and it can support the thread model with no
difficulty.
>
> > The problem is that at this point IMHO the product is designed to fit into
> > a client model and only very restrictive server models. I think basically
> > all we'll need to do is pull the crypto library and the protocol steps out
> > of the code and rewrite the protocol in a state engine where we simply
> > keep track of the progress of each connection.
>
> Agreed. I was under the impression that their existing state machine might
> be up to the task, though.
I have not looked at it yet. When I started going into the code I
immediately bogged down do to mainly lack of documentation. Having some
people who have and idea how the code works can rectify this situation.
BTW - I don't mind writing documentation... rather enjoy it actually.
>
> > The biggest difficulty I think will be understanding the present
> > architecture of the project. I've looked inside and what I saw were reams
> > and reams of code with no internal documentation to speak of.
>
> That was my impression, too. I suggest we try to understand the existing
> OpenSSL state machine, and come up with the tiniest possible bit of
> code that puts the 'right' kind of interface on the existing functionality.
> (Any bigger rewrite, and we'd never finish, I fear.)
> After we've gone down that path a ways, we might decide it's easier
> to rewrite parts of it from scratch, but let's put that off at least
> until we know what we're doing :-)
I usually find it easier to write from scratch. The size of the code
doesn't bother me - what does is the way memory is allocated and freed
(and sometimes not freed). There are ways to avoid memory leaks... it
involves good planning up front and disipline to not just allocate some
more memory in the bowles of the system because it is convienent. I think
the allocation issue will be the toughest to handle...
One should start with a structure that is allocated at the time the
connection is begun - with perhaps a preallocated pool - I like this idea
best - grab what you need all at once and if a little is wasted because a
"smaller" footprint is required by some algorithms - so what. I'm trying
to be reasonable here. If I need say 4K for the biggest (that is just a
guess) and the smallest takes say 2k then in my opinion it makes sense to
ask for the 4k in one go rather than say ten trips to malloc where
eventually you end up with a total of 2k. One way to see this is that if
malloc were to just hand the memory out of a pool- then you could
conceivably have the 2k you are given fragmented over 10 different pages
and meanwhile you find each physical page is 4k.... so the OS swaps in 40k
in total... (worst case of course... the memory manager can of course
allocate in one page increments and feed from there - but this means the
code must track how malloc does the allocation which means essentually
creating a memory page id that can be derived from the connection id and
passed into the memory subsystem.)
In any event - I have NOT looked at how openssl does this.
In a single connection situation it doesn't matter (generally - there may
be other things going on with the OS but the computer probably has gobs of
ram). In a threaded enviroment it doesn't _need_ to matter because a well
designed memory allocator will grab the thread number and create a free
space pool owned by the thread. The pool gets created when the thread is
created and destroyed when the thread dies and default allocations take
place in this pool (pool can shrink and grow pretty easy). Its nice and
clean and can control leaks as well.
For what you an I are discussing, I think we'd have to build a memory pool
manager (which needen't be hard) or alternatively watch really carefully
how we set up the data structures that carry the connection data nad
status. (this may be there - like I said - I haven't looked at this part
of the code).
In any event, if we design the proper underlying data structures to
accomodate what we want - then all we need to do is pass this data
structure into the proper state engine and the state engine can find the
parameters the low level subs need and call them. It can even do this in
a forked shared memory mode which is probably what we need. Either
forked or multithreaded will therefore work because any instance of the
low level functions will run in its proper memory and thus can run
concurrently and on multi processors if required.
----------
I have a more complex model in mind - I'd like to be able to take certain
functions - and fire them out the back door to dedicated servers. I know
that a fast processor can run a HUGE amount of decryption... but the
situation I'm looking at is a HUGE amount of demand... The server I
envision has to act as a front end running apache talking to a suite of
database servers running in parallel. In this situation the web server
for instance _might_ be able to handle the crypto - but not the crypto,
the database transactions and the code to dynamically create the web pages
all at the same time.
Since the database functions are already being exported, I want to export
the crypto as well and have the server basically just do the connections
and act as a traffic cop. This means that concevably we would end up
with the openssl code running behind a custom server in say 1-10 machines
each of which receives relayed connection requests from the front end. In
addition, since the databse machines may end up being I/O bound (likely)
it might make sense to offload the crypto deamons into the same machines
that run some of the database tasks - thus creating a tightly integrated
yet distributed group of processes that runs quite efficently and can be
scaled rather easily.
Now - there are probably a number of ways to handle this... via for
instance round robin dispatching to several servers with each handling
the whole job. But the idea here is to create parallelism right from the
moment the connection hits the server and I think it is a valid approach
and openSSL should be able to support it. Done properly, we can probably
once and for all control the damn memory leaks.
Terrell
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
