On Jul 1, 2005, at 4:59 PM, Fred Cox wrote:
--- Jim Davidson <[EMAIL PROTECTED]> wrote:
Fred,
Yup -- there are a few places in the code where
there's a Tcl API
without a corresponding C API. And, the docs are
out of date,
especially in the C API. Happily the code is
generally readable
although that's a crappy answer to the poor docs.
Thanks, then I haven't just been looking in the wrong
place. ;-)
Anyway, in this case we'd need a few things:
- An Ns_File (or some other name) object which
mapped to the
underlying File struct
- API's to get Ns_File's from an Ns_Conn.
Would this be an Ns_Set which gives the file contents,
similar to the result of Ns_ConnGetQuery(), keyed by
the form input name?
In a multipart upload, there's actually headers before each file
which get an Ns_Set. So, the C API could be more of an Ns_File
struct per file with pointer (or offfset) and length and an Ns_Set.
- API's to get offset, lengths, and copy the content
from the Ns_File
For each form input, I would be OK with a pointer to
the data and the length, with a warning that it should
be treated as read-only data. Reading down a bit, it
should also have a flag saying whether the data has
been written to a file, instead of being in memory.
First 4.0 doesn't support this file overflow stuff which you probably
don't need. It's in 4.5 and no parsing is done for you -- you just
get a NULL form, access to the fd, and an NS_CONN_FILECONTENT flag
bit and you're on your own. The reason is the parser is currently
designed to scan the content in one memory block -- this would need
to be changed (carefully). Most folks access content via simple the
simple Ns_Set interfaces so file based work (unless it simply mmap's
the result), would require something special anyway. For multipart
files which do fit in memory, there are some Tcl ns_conn options for
this in 4.0 and 4.5...
All that stuff is now just internal and accessed via
the ns_conn
command. I figured someone may want such API's
eventually but you're
the first to ask -- in general folks spend most of
their time with
the Tcl API's although the C API's are pretty good
in general. In
practice, the code wouldn't be so hard -- the docs
would probably be
more work.
I'd suggest you try out the Tcl API
My first impression was that even in Tcl, file uploads
were a bit creaky. The examples I saw could only
handle a single file upload per HTTP request, and
there was some indication that the file content had
been written to disk. Is there a better interface I
haven't found yet?
... including:
ns_conn files -- return list of file names
ns_conn fileoffset file -- offset for given file
ns_conn filelength file -- length
ns_conn fileheaders file -- headers
so something like:
foreach file [ns_conn files] {
set fp [file open $file.out w]
ns_conn copy [ns_conn fileoffset $file] [ns_conn filelength $file]
$fp
close $fp
ns_set print [ns_conn fileheader $file]
}
would copy file content to individual files and dump the headers to
the server log
-- it could be
used to handle all
the simple stuff and you can add commands to do any
weird stuff in
C. For completeness, I'll go ahead and add the
missing C API's but
they'll show up only in the head version of the
code, 4.5 or beyond,
and may be a few weeks or so as other changes are
backed up.
BTW: The aolserver I/O model is designed to
read-ahead all that data
before dispatching to the threads for processing --
if you're working
in a low-memory environment that may not be great
for you.
My particular application is being designed for a
reasonably high memory, but poor disk I/O system, so
this shouldn't be a major issue for me.
Then you should be ok and can get started with the Tcl code above or
a C API which is similar (but needs to be written).
There's
some other (undocumented) features to spool "large"
files to an open
temp file but then the various API's (in Tcl and C)
don't exist to
work with such files, only the Ns_ConnContentFd
function exists to
get at the open fd.
Is this in 4.0.10?
[EMAIL PROTECTED] find . -type f -exec egrep
Ns_ConnContentFd {} \; -print
NS_EXTERN int Ns_ConnContentFd(Ns_Conn *conn);
./include/ns.h
[EMAIL PROTECTED] dirs
~/aolserver/aolserver-4.0.10
Yup -- that API is new in 4.5, giving you access to the temp fd and
nothing else.
Factoring out all that so
in-memory and in-temp-
file would do the right thing would be quite a bit
more work to get
right.
What's the size limit that triggers this behavior? Is
it configurable? Web searching only brought up CVS
commits and a ChangeLog entry.
Core limits used before the request are known are configurable per-
driver -- you can see the code in nsd/driver.c leading to config
options such as:
ns_section ns/server/server1/modules/nssock
ns_param maxinput 1024000; total in-memory which is request,
headers, and content if it will fit
ns_param maxline 1024 ; length of a single request or header line
before content
Assuming requests and headers fit within maxinput, the server than
figures out the request and uses a "limits" object to manage
resources including total content upload (file or memory). These
limit objects are create and mapped to url's via the ns_limit's command:
ns_limit set 20meg -maxupload [expr 20 * 1024 * 1000]
ns_limit register 20meg POST /upload.adp
That code is in nsd/limits.c. The default limits are 10meg upload I
think. The ns_limit stuff is new but not yet documented.
In practice, if you aren't handling large files which would need the
overflow to temp file, than 4.0 is fine with the Tcl api's. If you
need to handle large temp files, than the new code in 4.5 is needed
plus your custom code to scan/parse the input. If you need the C
API's you need to wait for core changes in 4.5 or write some simple C
wrappers to Tcl which is a bit weird but done at times. I'd suggest
you try the Tcl stuff in 4.0 now -- it should get you started. I'll
add the C API when I can get to it in the next few weeks.
As for the docs, anyone interested in helping pull all this resource
management stuff together would be great -- an overview doc and/or
man pages would be useful. I know I should have written the docs
with the code but I failed -- sorry.
-Jim
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
