John, I think your analysis is correct. You have to avoid certain API or you do extra work, and maybe take up extra space. Usually this is unimportant, but in your case it isn't.
Anyway, sounds like you are making progress. Please let me know if I have led you down the wrong path. This stuff isn't really documented anywhere. tom jackson On Tue, Nov 24, 2009 at 2:44 PM, John Buckman <[email protected]> wrote: > Tom, I figured out the problem with the "memory bloat" when I used your > alternative ns_getform to parse a large file upload temp file. > > The problem is that if I call ns_queryget after calling your ns_getform, that > causes aolserver to re-parse the large file upload, and to do it in the old > memory-inefficient way. > > I'm not sure if there's a way for your version of ns_getform to tell the > aolserver internals that the parsing of the form is done, so subsequent > ns_queryget calls don't cause a re-parse. > > At any rate, there's an easy workaround, which is to read things right out > the ns_set that your ns_getform populates. > > -john > > > On Nov 23, 2009, at 8:47 PM, John Buckman wrote: > >> Tom, there is some sort of weird interaction effect on aolserver when doing >> tcl stuff and there is a large file as the temporary file. Some cases cause >> the nsd process to bloat to the size of the memory. I wasn't able to figure >> out why. I wasn't able to fix your ns_getform to not bloat. >> >> However, I was able to copy Vlad's ns_parseformfile proc for use on >> aolserver from naviserver, and it doesn't bloat. >> >> Here is the code for handling a large file uploaded as a file, rather than >> in memory: >> >> set type [ns_set iget [ns_conn headers] content-type] >> set form [ns_set create] >> ns_parseformfile $form $type >> array set formdata [ns_set array $form] >> puts "final array: [array get formdata]" >> >> >> proc ns_parseformfile { form contentType } { >> >> set fp [ns_conn contentchannel] >> >> if { ![regexp -nocase {boundary=(.*)$} $contentType 1 b] } { >> puts "Warning: no MIME boundary" >> return >> } >> >> fconfigure $fp -encoding binary -translation binary >> set boundary "--$b" >> >> while { ![eof $fp] } { >> # skip past the next boundary line >> if { ![string match $boundary* [string trim [gets $fp]]] } { >> continue >> } >> >> # fetch the disposition line and field name >> set disposition [string trim [gets $fp]] >> if { ![string length $disposition] } { >> break >> } >> >> set disposition [split $disposition \;] >> set name [string trim [lindex [split [lindex $disposition 1] =] 1] \"] >> >> # fetch and save any field headers (usually just content-type for >> files) >> >> while { ![eof $fp] } { >> set line [string trim [gets $fp]] >> if { ![string length $line] } { >> break >> } >> set header [split $line :] >> set key [string tolower [string trim [lindex $header 0]]] >> set value [string trim [lindex $header 1]] >> >> ns_set put $form $name.$key $value >> } >> >> if { [llength $disposition] == 3 } { >> # uploaded file -- save the original filename as the value >> set filename [string trim [lindex [split [lindex $disposition 2] >> =] 1] \"] >> ns_set put $form $name $filename >> >> # read lines of data until another boundary is found >> set start [tell $fp] >> set end $start >> >> while { ![eof $fp] } { >> if { [string match $boundary* [string trim [gets $fp]]] } { >> break >> } >> set end [tell $fp] >> } >> set length [expr $end - $start - 2] >> >> # create a temp file for the content, which will be deleted >> # when the connection close. ns_openexcl can fail, hence why >> # we keep spinning >> >> set tmp "" >> while { $tmp == "" } { >> set tmpfile [ns_tmpnam] >> set tmp [ns_openexcl $tmpfile] >> } >> >> catch {fconfigure $tmp -encoding binary -translation binary} >> >> if { $length > 0 } { >> seek $fp $start >> ns_cpfp $fp $tmp $length >> } >> >> close $tmp >> seek $fp $end >> ns_set put $form $name.tmpfile $tmpfile >> >> if { [ns_conn isconnected] } { >> ns_atclose "ns_unlink -nocomplain $tmpfile" >> } >> >> } else { >> # ordinary field - read lines until next boundary >> set first 1 >> set value "" >> set start [tell $fp] >> >> while { [gets $fp line] >= 0 } { >> set line [string trimright $line \r] >> if { [string match $boundary* $line] } { >> break >> } >> if { $first } { >> set first 0 >> } else { >> append value \n >> } >> append value $line >> set start [tell $fp] >> } >> seek $fp $start >> ns_set put $form $name $value >> } >> } >> close $fp >> } >> > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > <[email protected]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: > field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[email protected]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
