I think we talked about this before, but I can't find it in the
mailing list archive. Anyway, the problem with recording the upload
process is all the locking that's required. You could minimize this,
e.g. by only recording uploads above a certain size, or to a certain
URL.
Yes, that is true, but at the same time, GET requests do not carry the
body, so no locking willhappen. POST uploads are very slow processes in
a sense that user expects to wait until it finishes and locking will
happen only for uploads. It is possible further minimize it by doing
locking only several times instead of on every read but avoiding locks
is not possible.
It reminds me of a similar problem we had. Spooling large uploads to disk:
https://sourceforge.net/mailarchive/forum.php?thread_id=7524448&forum_id=43966
Vlad implemented the actual spooling, but moving that work into the
conn threads, reading lazily, is still to be done.
Lazy uploading is exactly the hook you need to track upload progress.
The client starts to upload a file. Read-ahead occurs in the driver
thread, say 8k. Control is passed to a conn thread, which then calls
Ns_ConnContent(). The remaining content is read from the client, in
the context of the conn thread and so not blocking the driver thread,
and perhaps the content is spooled to disk.
To implement upload tracking you would register a proc for /upload
which instead of calling Ns_ConnContent(), calls Ns_ConnRead()
multiple times, recording the number of bytes read in the upload
tracking cache, and saving the data to disk or wherever.
A lot more control of the upload process is needed, whether it be to
control size, access, to record stats, or something we haven't thought
of yet. If we complete the work to get lazy reading from the client
working, an upload tracker will be an easy module to write.
One problem i see with lazy uploads that if you have multiple clients
doing large POSTs, spawning multiple clients for a long time reading
that content will waste resources, each conn thread is heavy, with Tcl
interp. Using driver thread reading small chunks from the connections
and putting it into file will keep everything smooth. But with small
uploads on fast network this may not be an issue, so it needs a
compromize solution here, may be configurable options. Currently,
spooling into file can be disabled/enabled, lazy spooling may be
implemented similar way. Actually, lazy file spooling can be easily
done, because even Ns_ConnContent calls SockRead which does spooling, we
just need to introduce an option that tells how much we should spool in
the main thread and then continue in the conn thread.
--
Vlad Seryakov
571 262-8608 office
[EMAIL PROTECTED]
http://www.crystalballinc.com/vlad/
