Am 03.01.2006 um 01:07 schrieb Vlad Seryakov:
Will it be more generic way just to set upper limit, if we see that
upload exceeds that limit, pass control to conn thread and let it
finish reading, this way, even spooling to file will work, because
each upload conn thread will use their own file and will not hold
driver thread.
But we already have this limit. It is: maxreadahead.
If we'd want to kill many flies with one stoke, I believe
we will have to step out of the box!
By doing everything in the driver thread (provided we get the
AIO to work, which I'm sure can be done cross-platform) we are
getting only one of the things solved: upload of large conent.
We are not solving the upload progress nor any other security,
quota, whatelse, reqs. In order to do this, we'd need hooks into
the driver thread processing. This will inevitably lead to adding
a Tcl script processing into the driver thread plus some kind of
locking which could possibly cripple drivers performance.
I would not like to do anything in the driver thread which is Tcl-
related.
And, if any IO is being done there, it should not be blocking.
One of the solution: move everything which exceeds "maxreadahead" into
the connection thread. This way you have all bits and pieces and you
can define whatever checking/statistic policy you need. The code doing
the multipart dissasembly can have hooks to inspect for progress, you
can stop/discard the upload based on some constraints (file size,
whatelse)
so you have total control. The down-side: you engage the conn thread and
risk getting DOS attacks, as Gustaf already mentioned in one of his last
responses to this thread.
I think naviserver model using driver-thread for getting connections
accepted and their data read in-advance before passing ctrl to conn
thread
is good for most of the dynamic-content systems but not for doing
(large)
file uploads and/or serving (large) static content (images, etc).
I believe that event-loop-type (or AIO processing) is more suitable
for that.
After all, you can have 100's of slow peers pumping or pulling files
to/from
you at the snail speed. In such case it is a complete waste to
allocate that
many connection threads as it will blow the system memory. A
dedicated pool of
threads each running in the event-loop mode would be far more
appropriate.
Actually, just one thread in event-loop would do the work perfectly
on a single
CPU box. On many-CPU box a pool of threads (one per CPU) would be
more appropriate.
So, there it comes: we'd have to invent a third instance of the
processing!
1. driver thread
2. connection thread
3. spooling thread
The driver thread gets the connection and reads all up to maxreadahead.
It then passes the connection to the conn thread.
The conn thread decides to run that connection entirely (POST of a
simple form
or GET, all content read-in) OR it decides to pass the connection to the
spooling thread(s).
The spooling threads operate in event-loop mode. There has to be some
kind
of dispacher which evenly distributes the processing among spooling
threads.
Once in the spooling thread, the connection is processed entirely
asynchronously
as in Tcl event-loop. In fact, this whole thing can be implemented in
Tcl with
the building blocks we already have: channel-transfer between
threads, thread
management (create, teardown). Event the dispatcher can be done in
Tcl alone.
After processing in the spooling thread, the entire connection can
then be
bounced back to the connection thread OR finished in the spooling
thread.
I hope I did not write a tons of nonsense and thank you for being
patient
and reading up to here :-)
So, what do you think?
Zoran
