Re: [AOLSERVER] upload progress
> On 11/24/09 5:13 PM, John Buckman wrote: >> Is there any access (in C or Tcl) to an upload-in-progress in aolserver? > > It'd be nice if we extended ns_info with [ns_info driver ...] that could > give you connection-level info. from the driver thread. In its simplest > form, all we need is to expose the total bytes read/written on a socket > from the driver thread. Bytes read of the POST request's body and the > anticipated Content-Length enables us to compute a rough "progress" - > using the unique URL bit gives us an opaque handle to identify which > connection we're interested in. I've learned a few things by deploying a large-file-upload feature on aolserver: 1) IE times out on large file uploads over DSL, as does Chrome and Safari. Only Firefox seems to have a long enough timeout to enable 600mb file uploads over DSL. 2) All the other file upload sites use a client-side widget to upload a file in parts, not using the browser's upload feature at all. Then, they have a thin server-side program which accepts small chunks of the file upload at a time. Once the widget decides the entire file has been sent, it submits to a new web page, which then collects all the small file chunks. So... instead of working on an upload-in-progress feature, it would make sense instead to have a client-side widget (javascript/flash/java) that sends file upload chunks to a server-side tcl script, and then have a "harvester" tcl script once the widget says the file upload is complete. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Is this vulnerable to sql injection?
> I've been alerted that a site I maintain, running on AOLserver 4.5.0
> using the nspostgres driver, may be vulnerable to sql injection.
>
> A typical adp page performs a query like this:
>
> set sql_query "select * from sometable where entrynumber = $id"
>
> In a previous discussion thread here ("ns_db and bind variable
> support") I see "ns_db prepare..." mentioned. Is that a safer way to
> perform db queries in adp pages?
Yes, you're vulnerable to a SQL injection attack in the example above, if the
"id" variable is coming from the user. Most every web programming language has
the same vulnerability.
You need to quote-protect any input from a user to prevent this attack. I use:
> proc sqlquote {in} {
> regsub -all ' $in '' out
> return $out
> }
otherwise the user can put an end-quote in the input they send you, and then
put their own sql statement in to execute. It's a very simple attack to perform.
ie your code should look like this:
> set sql_query "select * from sometable where entrynumber = [sqlquote $id]"
Note that SQLlite has built-in Tcl support that does this automatically, which
is very convenient.
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] large file uploads
> I think your analysis is correct. You have to avoid certain API or > you do extra work, and maybe take up extra space. Usually this is > unimportant, but in your case it isn't. > > Anyway, sounds like you are making progress. Please let me know if I > have led you down the wrong path. This stuff isn't really documented > anywhere. Your code works great and you didn't lead me down any wrong paths. As long as no calls to ns_queryget are made when a large file upload is received, then your form parser grabs the files out of the form in a memory efficient way. I'm just about to put it into widespread production, but it's been working on my production linux server and osx development machines perfectly so far. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
> We are using AOL 4.5.1 (OpenACS) in production environment with many file
> uploads (and downloads)
> and the memory usage increases continously. We begin with near 4 GB RAM and
> the this increase to
> near 11 GB of virtual memory and we have to restart the server. When the
> server is over 6 GB of
> memory, the execs (sendmail, unzipos, ...) can not be launched. Fork
> problem. I will like find any
> solution to that mem increase.
>
> We have developed a small ns library (using mkZiplib1.0) that we have named
> nsunzip, to avoid the execs
> of unzip. We plan add too zip action. If anybody are interested I can send it.
Agustin, there are a few things you should do which will solve your problem.
First, I recommend having fewer threads running, as that will lower your bloat
significantly. I run with 10.
Then, switch to using temporary files (instead of memory), by changing your
config to:
> ns_section "ns/server/$server1/module/nssock"
> ns_param maxinput1024
and at the bottom:
> ns_limits set default -maxupload "204800"
that will get rid of the nsd process bloat while the file is uploading. You'll
still have bloat when the file is unzipped, which you can solve by removing any
mention of ns_queryget when you receive the upload, and replacing it with Tom's
ns_getform (posted here a few days ago) and then reading the ns_set that
process returns.
After these changes, my nsd process is stable at 16mb of RAM, despite receiving
800mb zip uploads that I unzip inside aolserver.
Hope that helps!
-john
## ns_getform made saf(er)
# An improved version would return the open fp instead of the tmpfile,
# or better use a system call which returns an fp
rename ns_getform ns_getform_unsafe
#
# ns_getform --
#
# Return the connection form, copying multipart form data
# into temp files if necessary.
#
proc ns_getform {{charset ""}} {
global _ns_form _ns_formfiles
#
# If a charset has been specified, use ns_urlcharset to
# alter the current conn's urlcharset.
# This can cause cached formsets to get flushed.
if {$charset != ""} {
ns_urlcharset $charset
}
if {![info exists _ns_form]} {
set _ns_form [ns_conn form]
foreach {file} [ns_conn files] {
set off [ns_conn fileoffset $file]
set len [ns_conn filelength $file]
set hdr [ns_conn fileheaders $file]
set type [ns_set get $hdr content-type]
set fp ""
while {$fp == ""} {
set tmpfile [ns_tmpnam]
set fp [ns_openexcl $tmpfile]
}
fconfigure $fp -translation binary
ns_conn copy $off $len $fp
close $fp
ns_atclose "ns_unlink -nocomplain $tmpfile"
set _ns_formfiles($file) $tmpfile
#ns_set put $_ns_form $file.content-type $type
# NB: Insecure, access via ns_getformfile.
#ns_set put $_ns_form $file.tmpfile $tmpfile
}
}
return $_ns_form
}
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] upload progress
> the typical meachnism for upload progress bars is that separate (ajax) queries > are used to query the state of an running upload, which is identified by some > unique ID (e.g. X-Progress-ID in > http://wiki.nginx.org/NginxHttpUploadProgressModule, > or some other heuristics. e.g. based on URL and peer addr). > So, one needs just a thread-safe introspection meachnism, which > checks in the spooler the state (such as "ns_upload_stats" > in naviserver) and returns it back. The spooler hast just to update > these statistics when it recieves data. Exactly - what is needed in aolserver is simply a tcl command for introspection, specifically how many bytes have been received in some particular case. That data must be somewhere, what I was wondering is if anyone had an idea *where*. -j -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
Tom, I figured out the problem with the "memory bloat" when I used your
alternative ns_getform to parse a large file upload temp file.
The problem is that if I call ns_queryget after calling your ns_getform, that
causes aolserver to re-parse the large file upload, and to do it in the old
memory-inefficient way.
I'm not sure if there's a way for your version of ns_getform to tell the
aolserver internals that the parsing of the form is done, so subsequent
ns_queryget calls don't cause a re-parse.
At any rate, there's an easy workaround, which is to read things right out the
ns_set that your ns_getform populates.
-john
On Nov 23, 2009, at 8:47 PM, John Buckman wrote:
> Tom, there is some sort of weird interaction effect on aolserver when doing
> tcl stuff and there is a large file as the temporary file. Some cases cause
> the nsd process to bloat to the size of the memory. I wasn't able to figure
> out why. I wasn't able to fix your ns_getform to not bloat.
>
> However, I was able to copy Vlad's ns_parseformfile proc for use on aolserver
> from naviserver, and it doesn't bloat.
>
> Here is the code for handling a large file uploaded as a file, rather than in
> memory:
>
> set type [ns_set iget [ns_conn headers] content-type]
> set form [ns_set create]
> ns_parseformfile $form $type
> array set formdata [ns_set array $form]
> puts "final array: [array get formdata]"
>
>
> proc ns_parseformfile { form contentType } {
>
> set fp [ns_conn contentchannel]
>
>if { ![regexp -nocase {boundary=(.*)$} $contentType 1 b] } {
> puts "Warning: no MIME boundary"
>return
>}
>
>fconfigure $fp -encoding binary -translation binary
>set boundary "--$b"
>
>while { ![eof $fp] } {
> # skip past the next boundary line
> if { ![string match $boundary* [string trim [gets $fp]]] } {
> continue
> }
>
> # fetch the disposition line and field name
> set disposition [string trim [gets $fp]]
> if { ![string length $disposition] } {
> break
> }
>
> set disposition [split $disposition \;]
> set name [string trim [lindex [split [lindex $disposition 1] =] 1] \"]
>
> # fetch and save any field headers (usually just content-type for files)
>
> while { ![eof $fp] } {
> set line [string trim [gets $fp]]
> if { ![string length $line] } {
> break
> }
> set header [split $line :]
> set key [string tolower [string trim [lindex $header 0]]]
> set value [string trim [lindex $header 1]]
>
> ns_set put $form $name.$key $value
> }
>
> if { [llength $disposition] == 3 } {
> # uploaded file -- save the original filename as the value
> set filename [string trim [lindex [split [lindex $disposition 2] =]
> 1] \"]
> ns_set put $form $name $filename
>
> # read lines of data until another boundary is found
> set start [tell $fp]
> set end $start
>
> while { ![eof $fp] } {
> if { [string match $boundary* [string trim [gets $fp]]] } {
> break
> }
> set end [tell $fp]
> }
> set length [expr $end - $start - 2]
>
> # create a temp file for the content, which will be deleted
> # when the connection close. ns_openexcl can fail, hence why
> # we keep spinning
>
> set tmp ""
> while { $tmp == "" } {
> set tmpfile [ns_tmpnam]
> set tmp [ns_openexcl $tmpfile]
> }
>
> catch {fconfigure $tmp -encoding binary -translation binary}
>
> if { $length > 0 } {
> seek $fp $start
> ns_cpfp $fp $tmp $length
> }
>
> close $tmp
> seek $fp $end
> ns_set put $form $name.tmpfile $tmpfile
>
>if { [ns_conn isconnected] } {
> ns_atclose "ns_unlink -nocomplain $tmpfile"
>}
>
> } else {
> # ordinary field - read lines until next boundary
> set first 1
> set value ""
> set start [tell $fp]
>
> while { [gets $fp line] >= 0 } {
> set line [string trimright $line \r]
> if { [string match $boundary* $line] } {
> break
> }
>
[AOLSERVER] upload progress
Naviserver has a very nice feature that allows (via javascript) to show a user the percent upload progress of a file. I tried porting their progress.c file to aolserver, but it's a significant effort, as it depends on other changes naviserver has implemented to the aolserver code. However, I was wondering if there wasn't a fairly simple way to implement something similar on aolserver. What I'd need is a way to know how much data has been uploaded. The way naviserver does it, is by asking you to POST your upload to a unique URL, and then they provide a tcl command that returns how many bytes have been uploaded to that unique URL. A javascript regularly polls an ADP page that uses the tcl progress command to tell you the number of bytes uploaded. Is there any access (in C or Tcl) to an upload-in-progress in aolserver? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
Tom, there is some sort of weird interaction effect on aolserver when doing tcl
stuff and there is a large file as the temporary file. Some cases cause the
nsd process to bloat to the size of the memory. I wasn't able to figure out
why. I wasn't able to fix your ns_getform to not bloat.
However, I was able to copy Vlad's ns_parseformfile proc for use on aolserver
from naviserver, and it doesn't bloat.
Here is the code for handling a large file uploaded as a file, rather than in
memory:
set type [ns_set iget [ns_conn headers] content-type]
set form [ns_set create]
ns_parseformfile $form $type
array set formdata [ns_set array $form]
puts "final array: [array get formdata]"
proc ns_parseformfile { form contentType } {
set fp [ns_conn contentchannel]
if { ![regexp -nocase {boundary=(.*)$} $contentType 1 b] } {
puts "Warning: no MIME boundary"
return
}
fconfigure $fp -encoding binary -translation binary
set boundary "--$b"
while { ![eof $fp] } {
# skip past the next boundary line
if { ![string match $boundary* [string trim [gets $fp]]] } {
continue
}
# fetch the disposition line and field name
set disposition [string trim [gets $fp]]
if { ![string length $disposition] } {
break
}
set disposition [split $disposition \;]
set name [string trim [lindex [split [lindex $disposition 1] =] 1] \"]
# fetch and save any field headers (usually just content-type for files)
while { ![eof $fp] } {
set line [string trim [gets $fp]]
if { ![string length $line] } {
break
}
set header [split $line :]
set key [string tolower [string trim [lindex $header 0]]]
set value [string trim [lindex $header 1]]
ns_set put $form $name.$key $value
}
if { [llength $disposition] == 3 } {
# uploaded file -- save the original filename as the value
set filename [string trim [lindex [split [lindex $disposition 2] =]
1] \"]
ns_set put $form $name $filename
# read lines of data until another boundary is found
set start [tell $fp]
set end $start
while { ![eof $fp] } {
if { [string match $boundary* [string trim [gets $fp]]] } {
break
}
set end [tell $fp]
}
set length [expr $end - $start - 2]
# create a temp file for the content, which will be deleted
# when the connection close. ns_openexcl can fail, hence why
# we keep spinning
set tmp ""
while { $tmp == "" } {
set tmpfile [ns_tmpnam]
set tmp [ns_openexcl $tmpfile]
}
catch {fconfigure $tmp -encoding binary -translation binary}
if { $length > 0 } {
seek $fp $start
ns_cpfp $fp $tmp $length
}
close $tmp
seek $fp $end
ns_set put $form $name.tmpfile $tmpfile
if { [ns_conn isconnected] } {
ns_atclose "ns_unlink -nocomplain $tmpfile"
}
} else {
# ordinary field - read lines until next boundary
set first 1
set value ""
set start [tell $fp]
while { [gets $fp line] >= 0 } {
set line [string trimright $line \r]
if { [string match $boundary* $line] } {
break
}
if { $first } {
set first 0
} else {
append value \n
}
append value $line
set start [tell $fp]
}
seek $fp $start
ns_set put $form $name $value
}
}
close $fp
}
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] large file uploads
Tom, thanks for the help! Setting maxinput as per: > ns_section "ns/server/$server1/module/nssock" > ns_param maxinput1024 does indeed avoid memory bloat during the large file upload. I used your safer ns_getform: it works fine, However, your ns_getform causes the nsd process to grow to the size of the uploaded file. I can't figure out what it is in your code that uses lots of memory. Any idea? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
On Nov 19, 2009, at 8:56 AM, Tom Jackson wrote: > There is a configuration setting which saves posted files to disk. > You need the ns_limits maxupload to be large enough, then maxinput > (not sure which config section) sets an upper limit for in memory > data. Yes, the naviserver people talked about that, but the problem is that then you get the raw data, that you need to do something with, rather than temporary files. I'm currently trying out naviserver, as they do seem to have solved the large-file-upload problem that aolserver has. It's working for me, and nsd process size is staying at 50mb even with multiple gig file uploads being handled. The main thing that naviserver has, that would probably work simply on aolserver, is a ns_parseformfile function, which parses the raw upload data in a memory efficient way. Here is what the code looks like: set tmpfile [ns_conn contentfile] ns_parseformfile $tmpfile $form [ns_set iget [ns_conn headers] content-type] array set formdata [ns_set array $form] It might straightforward to copy the "ns_conn contentfile" and "ns_parseformfile" functions to aolserver to solve this problem. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
On Nov 18, 2009, at 4:34 PM, Hossein Sharifi wrote: > Are you using AOLserver 4.5x on a 64-bit platform? nsd will crash on 32-bit > platforms when memory usage (or log files) exceed 2GB. I had to upgrade for > a similar issue involving memory usage. > > Although I think that it would be ideal to port the Naviserver code at some > point. No, I'm on a 32bit linux. I guess the solution is to upgrade to a 64bit OS and have way more memory than the max file size. I built naviserver today, with their spooler thread, and tested large file uploads. Naviserver is very efficient at memory when handling the file upload, but it does hold the entire uploaded file in memory as it hands the file off to my form handler. So, it has the same peak memory usage as aolserver, though it needs the peak memory for less time. Still, not an optimal solution. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] large file uploads
I'm developing a music submission system for Magnatune.com using aolserver and I'm seeing some problems with large file uploads. Specifically, aolserver periodically crashes due to a malloc error. It looks like aolserver stores files being uploaded (via form enctype="multipart/form-data") in memory as they're being uploaded. When people are uploading zip files of WAV-quality rips of CDs, that's 800mb of RAM per upload. A few concurrent uploads, and splat. It looks like Naviserver has changed the aolserver code in Naviserver 4.99.1 to > - Using temporary files for large uploads when content exceeds pre-configured > maxsize parameter > - New spooler thread for handling large file uploads Does it make sense to port this naviserver code over to aolserver? Or, is there another (easier) way to handle large file uploads that doesn't use lots of memory? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] can someone make this change
On 2008.04.09, Tom Jackson <[EMAIL PROTECTED]> wrote: You shouldn't be running configure directly! Please read the README file for how to install AOLserver. If you run ./configure, this is the type of error you should expect. I wouldn't go so far as to say "you shouldn't run configure directly" as others have pointed out, there are settings you can set via configure but not through nsconfig.tcl, but ... _if_ you run configure, you have to know to pass the TCLSH= to it: I'm sorry if I misunderstood, but I'm trying to get a 64bit version of aolserver built on MacOS 10.5, and my notes on building-for-64bit were that ./configure needed to be used (--enable-64bit). Lots of people posted problems with build aolserver on MacOS 10.5, so I was going with what I knew. Also, it looks to me like ./configure works perfectly well, except for those util/*.tcl scripts failing during "make install", and I have a 64bit version of aolserver (and Tcl 8.5 and Berkeleydb) running now, though building tdom 0.8.2 is still giving me trouble. FYI the ActiveState distribution of Tcl for MacOS is 32bit. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] can someone make this change
John Buckman schrieb: the utils/*.tcl files in CVS all need: #!/usr/local/bin/tclsh prepended at the top. Currently, they don't have this, and thus are run as shell scripts. hmm, shouldn't this be a "/usr/bin/env tclsh" instead? This looks to be the way it should be done (from tests/new/all.tcl) $ more tests/new/all.tcl #!/bin/sh # the next line restarts using tclsh \ exec tclsh "$0" "$@" -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] can someone make this change
Can someone make this change in CVS: the utils/*.tcl files in CVS all need: #!/usr/local/bin/tclsh prepended at the top. Currently, they don't have this, and thus are run as shell scripts. They also need their permissions to be executable. I know we've switched to a tcl-building method, but "./configure" still (mostly) works, but more importantly, those util scripts are used by ns.mak and modules depend on them to work. Dossy: if you are willing to give me cvs write privs, I can fix this. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Compression
On Apr 8, 2008, at 1:54 PM, Brett Schwarz wrote:
So, from my reading, its seems as though that in 4.5, we got on the
fly ADP page compression. However, it seems as though there isn't
comparable functionality for Tcl files (i.e. ns_return). I know
there are some work arounds, but I was just curious why this was
not implemented in the core? I took a look at the sources, and it
looks fairly straight forward to add this functionality. I see
there are three possibilities:
1) modify ns_return to take a flag. The optional connid parameter
makes parsing the args a little trickier.
Current:
ns_return ?connid? status type string (is connid ever used??)
I use:
ns_return_gzipped $conn $x
which is this, and works very reliably:
proc ns_return_gzipped {conn html} {
if {[string first {gzip} [ns_set get [ns_conn headers] {Accept-
Encoding}]] == -1} {
ns_return $conn 200 text/html $html
return TCL_OK
}
# john 9/21/07 -- found that we need to convert to utf-8, it isn't
automatic
set zl [ns_zlib gzip [encoding convertto utf-8 $html]]
set h [subst {HTTP/1.0 200 OK
MIME-Version: 1.0
Server: MoochServer/4.5.0a
Connection: close
Content-Type: text/html; charset=utf-8
Content-Length: [string length $zl]
Content-Encoding: gzip
$zl}]
ns_write $conn $h
return TCL_OK
}
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Compiling 4.5 on Mac OS X 10.5 leopard issues
I suspect the problem might have been that you didn't enable threads on tcl. I've never had any problems building aolserver on OSX 8.4.x on powerpc or intel. But, you know that now... -john On Mar 22, 2008, at 8:44 PM, Mark Aufflick wrote: Thanks Gustaf, I had trouble with cvs head earlier, but I have tried again. Here is what worked for me: 1. Compile/install of tcl 8.4.18 with --enable-threads ** 2. co aolserver cvs head 3. autoconf (being sure that ../tcl symlinks to my tcl 8.4.18 sources) 4. /opt/aolserver/bin/tclsh8.4 nsconfig.tcl -debug 5. sudo make install 6. Apply the pools.tcl default patch as per previous email thread (although probably not necessary) 7. Change config port to 8003 (Leopard seems strangely reluctant to respond to requests on port 8000) Now I get the welcome page. Thanks everyone. Mark. ** Notably the aolserver build failed if I compiled tcl with --enable-symbols=all for maximum debug symbols. The error was: Undefined symbols: "_Tcl_GetMemoryInfo", referenced from: _DumperThread in nsthreadtest.o This sounds like an issue with the tcl build process rather than aolserver though. On Sun, Mar 23, 2008 at 4:48 AM, Gustaf Neumann <[EMAIL PROTECTED] wien.ac.at> wrote: Dear Mark, this problem was discussed earlier here in the mailing list. I have commited a fix to the head version in the sourceforge cvs for the problem, which changes the semantics for "setrlimit" for compiles on Leopard and above. It passes now in case of unlimited number of open files OPEN_MAX files to setrlimit. -gustaf neumann Mark Aufflick schrieb: Anyhoo - once built, nsd segfaults, predictably in the dynamic loading stage: ... I get this even if I just try nsd --help I need some --help :) It seems that people have this running on Intel ok, and I have compiled Aolserver 4.x on earlier Mac OS X versions ok, but the huge string of defines to gcc has me a bit stumped about where to look. Thanks. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- Mark Aufflick contact info at http://mark.aufflick.com/about/contact -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] wiki very slow
FYI, the new wiki server is timing out on me: Safari could not open the page “http://dev.aolserver.com/wiki/” because the server stopped responding. whereas the old location (thanks) loads quickly: http://panoptic.com/wiki/aolserver And sorry to be a complainer, but the 7 second page load times you improved it to sound awfully long as well (but you're right, at least useable). -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] wiki very slow
Just an FYI that the Aolserver wiki link from www.aolserver.com: http://dev.aolserver.com/Tcl_API takes over 20 seconds to load a page, and has been like this for a few weeks. It's not really useable at this speed. It was fine at the previous host. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver 64bit
On Feb 29, 2008, at 6:03 PM, Tom Jackson wrote: I've done load testing with 64bit, watching the memory footprint as I go, never grew for me with 100k/test. It could be that the 64bit uses more memory, and you just never noticed that your application grew in memory footprint over time and then stablized. The best way to test it is to do load tests for particular packages and watch memory, it shouldn't take long to see if it keeps growing while hitting a single page. But multiple pages, which simulates real world loads will usually grow. I do that. I've run 100 simultaneous copies of wget in its "crawl" mode (where it follows links) and after 24h I had no process growth. I suspect that my bloat is actually a tcl thread holding a large data structure in memory, and then when it exits, the OS keeping hold of that memory in that thread to avoid semaphore locking if the memory went back to the general pool. I've seen that behavior on Linux and Windows. So, lots of Tcl threads, all having run at some large memory footprint size at some time, that would cause bloat. -john $ uname -a Linux localhost 2.6.18-gentoo-r4 #1 SMP Mon Nov 20 16:49:16 UTC 2006 x86_64 AMD Turion(tm) 64 X2 Mobile Technology TL-52 AuthenticAMD GNU/Linux tom jackson On Friday 29 February 2008 09:34, John Buckman wrote: I my case... My only desire would be getting AOLServer to work perfectly on 64bit platforms. I have not heard of anyone that has AOLServer working on it with a decent load (more than 10K hits / day) on a 64 bits platform and do not restart it for at least 1 month. Aolserver/64bits works perfectly for me at BookMooch.com, I run it entirely on a single 8 core server with 24bg of RAM, running debian. I'm doing a bit less than a half-million hits a day, and every one of those hits is a dynamically created tcl page, with database operations (my images are served from a separate, dumb http server) There are some minor compilation issues on 64 bit, that would take 1/2 a day to permanently solve in the cvs tree (Dossy helped me compile mine, in my case) but otherwise it's 100% stable. My uptime is about 2 months. I sometimes have to reboot due to a semaphore deadlock, but I think that's in the database library, not aolserver (it happens during my nightly full backup). -john www64:~# uname -a Linux www64 2.6.18-5-amd64 #1 SMP Thu Aug 30 01:14:54 UTC 2007 x86_64 GNU/Linux www64:/usr/local/aolserver/servers/bookmooch/modules/nslog# wc -l access.log.001 432861 access.log.001 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Google Summer of Code
On Fri, Feb 29, 2008 at 05:41:44PM +0100, Juan Jos? del R?o [Simple Option] wrote: In my case I run AOLServer with customized code on top of it. No OpenACS. No modules except nspostgres. In 32 bits it works like a charm. No memory leaks. It simply works (damn fast!). Hm, so you see memory leaks when compiled for 64 bit, but none when compiling the same code 32 bit - interesting. Can you reproduce the leakage under Valgrind? FYI, I see no leaks under 64bit, but I do see a fair amount of "bloat". On 32bit Linux, my nsd process ran around 1.2gb, while under 64bits I'm around 7gb. However, no leaks. After a few days my nsd process size does not continue to grow, it is stable at ~17gb (I have a 10gb BerkeleyDB cache in process). I'm not clear on why anyone would need 64bit Aolserver unless they needed to use more than 2gb of RAM. I'm also not clear that the bloat is aolserver, and not tcl or other libraries I'm using. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] aolserver 64bit
I my case... My only desire would be getting AOLServer to work perfectly on 64bit platforms. I have not heard of anyone that has AOLServer working on it with a decent load (more than 10K hits / day) on a 64 bits platform and do not restart it for at least 1 month. Aolserver/64bits works perfectly for me at BookMooch.com, I run it entirely on a single 8 core server with 24bg of RAM, running debian. I'm doing a bit less than a half-million hits a day, and every one of those hits is a dynamically created tcl page, with database operations (my images are served from a separate, dumb http server) There are some minor compilation issues on 64 bit, that would take 1/2 a day to permanently solve in the cvs tree (Dossy helped me compile mine, in my case) but otherwise it's 100% stable. My uptime is about 2 months. I sometimes have to reboot due to a semaphore deadlock, but I think that's in the database library, not aolserver (it happens during my nightly full backup). -john www64:~# uname -a Linux www64 2.6.18-5-amd64 #1 SMP Thu Aug 30 01:14:54 UTC 2007 x86_64 GNU/Linux www64:/usr/local/aolserver/servers/bookmooch/modules/nslog# wc -l access.log.001 432861 access.log.001 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Google Summer of Code
On 2008.02.28, Matthew M. Burke <[EMAIL PROTECTED]> wrote: I am convinced we could attract some students, but I don't want to commit unless there's at least a little more positive response. Another possibility is that I know Clif Flynt, Jeff Hobbs and other Tcl folks are putting together an application. So perhaps the better approach would be to list some AOLserver-related projects with their application. What do folks think? I think the silence from the community speaks volumes, unfortunately. I'm not (currently) a Javascript programmer but I thought Dossy's proposal to make Javascript a 1st-class language in Aolserver was by far the most bang-for-the-buck, and most likely to restart interest in aolserver. And if aolserver had serious javascript, I'd probably learn javascript and put a lot of my own time into making it work right. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] all interps reload?
Can you describe what you are trying to achieve and how you test to see that it isn't working? It would be nice to see some working code/ configuration to supplement the documentation. Quite simply, I prefer aolserver interps to load the source once, keep it in memory in parsed form. This makes page load times fast. My .adp files tend to be mostly-empty shells that call .tcl procs, so that the procs are parsed and in memory already. On my development machine, I have aolserver set to re-source all my code for each page load. That's a lot slower, but it makes development easier. On a production machine, I would like to "push up" a new web site software version w/o restarting the web server. Make sense? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] all interps reload?
On Nov 2, 2007, at 3:56 PM, Rusty Brooks wrote: I use ns_eval, which can be used to run a command in all interpreters. Honestly, though, I don't fool around with packages much with aolserver... I mostly have directories, and a proc that will recursively go through them, starting at a given root. So if I have Thanks Rusty, that worked well for me. This works: ns_eval -sync lib_reload_all where "lib_reload_all" is a custom proc to resource everything. Though I would have thought this would work, it didn't work for me: ns_eval ns_markfordelete -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] all interps reload?
What do the other aolserver programmers do to ask all the tcl interps
to reload, so that new code can be brought in without restarting the
server?
I have a technique (described below) but it's not bug-free, so I was
wondering what people recommended?
One possibility, would be to ask all the tcl interpreters to exit on
the next page request, so that a new thread and thus new tcl
interpreter is created, loading up the new libraries. I don't know
how to do that, though.
Currently, I have a namespaced global variable $reload::time that is
set to the time when the last "package forget/package require". was
run When I want to reload all interpreters, I set a nsv variable to
the current time, and a function registered as
ns_register_filter postauth GET /* mooch_reload_check
runs for every page, so that if the nsv's reload time is not the same
as the local interp's reload time, then I run:
foreach p [package names] {
package forget $p
}
This seems to mostly work, but some interpreters don't play nice.
Other suggestions ?
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] UTF/asian charset in urls
On Oct 1, 2007, at 8:52 AM, Hossein Sharifi wrote: I don't see this problem in 4.0 or 4.5. It looks like searching for 秋 on bookmooch first goes to http://www.bookmooch.com/search?w=%E7%A7%8B&search.x=14&search.y=13 (which looks fine - that's the correct URL encoding of the UTF-8 representation of that character) but that page immediately redirects to a cleaner search URL ( http://bookmooch.com/m/s/...) which contains the $map(...). Maybe it's related to the code that builds the cleaner URL? Thanks Hossein, for the insight. You were right, my problem was caused by a bug in the ncgi url encoding function. ncgi builds a $map() array of character conversions, and puts $map (character) around anything that isn't A-Za-Z0-9, then a [subst - nocommand $string] around all that. The higher-UTF characters cause problems with the ncgi function, and they emerge from it with $map() wrapped around them. A simple regexp to remove $map() from what ncgi can't encode, and now it works perfectly. I have to say that the aolserver handling of UTF is really well done. At Lyris, we never did quite get the UTF handling in TclHttpd done perfectly, there were still some fringe cases that caused garbling. With aolserver, except for this ncgi problem, and figuring out that I needed to switch to utf-8 as the default (from the default iso 8859-1), non-english character sets have worked perfectly. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] code for 404 pattern (aka static cache)
On Sep 29, 2007, at 11:00 PM, Jeff Rogers wrote: John Buckman wrote: Bas Scheffers was kind to share his 404-pattern code with me (ie, a custom 404 handler to enable static caching of frequently requested files), which I used to write my own. It's not rocket science, but since I asked the question, I thought I'd share my code, in case anyone else finds it helpful. I've taken the liberty of adding this to the wiki, on the "Cookbook" page: http://panoptic.com/wiki/aolserver/AOLserver_Cookbook Thanks for reminding us of this wiki page. I'll endeavor to put my "bits" in there in the future. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks
When comparing lighthttpd vs aolserver, notice that aolserver only does worse than lighthttpd for large files, and on the same file system/hardware. Thus, the difference in benchmarks is not likely to be the access logs or disk. Lighthttpd is *not* using the system call to send a file to a socket (I forget the name) as this call was taken out of the Linux kernel, I believe with 2.4. I remember reading a note about this from Linus, that the performance for that system call was terrible, so they were taking it out. Based on my own experience with sending large files over tcpip, the difference is that aolserver uses a thread-based approach vs a lighthttpd's single-thread async approach. But really, I'm not sure aolserver's performance is really an issue anyhow, because aolserver only slows down with large files (100mb+) and at that point, you're completely saturating your network connection (in my 128mb benchmark below, aolserver is sending 695mb per second!) So, I don't see any point in worrying about improving aolserver's plain-file-sending, at least until we get start getting 10 gigabit network cards (!) -john On Sep 27, 2007, at 4:53 PM, Tom Jackson wrote: I was looking at lighttpd performance (at http://trac.lighttpd.net/trac/wiki/Docs%3APerformance ) Considering how well AOLserver stands up to lighttpd, my question was why does lighttpd do better? For sending small files AOLserver is slightly better, but then performance goes down. One reason might be that lighttpd uses a syscall that sends the file directly to the network adapter, bypassing lighttpd. I wonder if AOLserver does this, or if it is possible to detect errors in the webserver if the whole file isn't sent. Another reason might be that lighttpd doesn't write an access.log by default. I wonder if you can turn this off in AOLserver? Or, if not, can the benchmark be run for lighttpd with the access.log enabled: http://trac.lighttpd.net/trac/wiki/Docs%3AModAccessLog With multi-cpus and lighttpd processes, lighttpd might mess up the access.log writing, but if these writes are taken out of the performance equation, it sure gives lighttpd a headstart. Are there any hints on how to setup the benchmarking? I can run one here on a 1cpu/2core/64bit laptop. If we can establish a benchmarking suite, it would help test various setting for ns_limits and ns_pools. It would also be interesting to see benchmarks under misbehaving clients. One other factor is number of disks. Lighttpd recommends 2 x number of disks to determine processes, although it seems like this wouldn't be such an easy choice. Ideas? tom jackson On Tuesday 25 September 2007 00:50, John Buckman wrote: I did some benchmarks of aolserver vs lighthttpd on plain files of various sizes, on my 8cpu 64 bit server. For 3 runs, with a 4k text file, the lighthttpd stats were 15103.87/ s, 14845.20/s and 15307.17/s, vs (as Dossy reports http://dossy.org/ archives/000517.html) aolserver's 15237.00/s. Result: Aolserver is performing virtually identically to lighthttpd with a small 4k file. With a 44k JPG (the BookMooch home page illustration) aolserver is slightly slower: aolserver: 8164.54/s, 8283.93/s lighthttpd: 10281.01/s, 9969.91/s With 1,414k sized zip, aolserver is about 1/2 as fast: aolserver: 406.36/s, 421.10/s lighthttpd: 844.05/s, 807.83/s with a 128mb file, aolserver is 40% slower: aolserver: 5.42/s lighthttpd: 8.88/s And just FYI "hello world" in tcl (<% ns_adp_puts "hello" %>): aolserver: 950.85/s I haven't done a "hello world" in C: I assume it'd be close to the 16k/s speed -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] code for 404 pattern (aka static cache)
Bas Scheffers was kind to share his 404-pattern code with me (ie, a
custom 404 handler to enable static caching of frequently requested
files), which I used to write my own.
It's not rocket science, but since I asked the question, I thought
I'd share my code, in case anyone else finds it helpful.
-john
from config.tcl:
ns_section "ns/server/$server1/redirects"
ns_param 404 /404.adp
$ cat 404.adp
<% handle_404 %>
proc handle_404 {} {
set myurl [myurl]
set parts [split $myurl /]
set root [lindex $parts 1]
if {$root == "photo"} {
return [404_photo $parts]
} else {
ns_returnredirect "[myhost]/"
return
}
}
proc 404_photo {parts} {
set userid [file rootname [lindex $parts 2]]
set d [user_photo_data $userid]
if {$d == ""} {
set fn "/b/photo/unknown.jpg"
} else {
set fn "/b/photo/${userid}.jpg"
write_binary_file $fn $d
}
ns_returnfile 200 "image/jpeg" $fn
}
proc myurl {} {
return [lindex [split [ns_conn request]] 1]
}
proc photo_cache_dirty {userid} {
set fn "/b/photo/${userid}.jpg"
file delete $fn
}
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
[AOLSERVER] UTF/asian charset in urls
Does anyone know what these "$map(%E7%A7%8B)" things are in URLs when asian-UTF characters are used? Ie, typing into a search form on aolserver (BookMooch) for the japanese character "秋" converts it to http://bookmooch.com/m/s/$map(%E7%A7%8B) That $map looks like an array lookup, but I've never seen any documentation about this, or how it might be re-converted back into UTF. I can't find any mention of $map in the aolserver or tcl source code, nor online. The other possibility is that the hex codes: %E7%A7%8B are just 3- byte representations of each character, in linear order, that needs to be reconverted back into UTF somehow. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks
On Sep 28, 2007, at 8:43 AM, John Buckman wrote: My solution to that problem was simply caching in the filesystem and serving static files. The way this works in a multi-server environment is that the custom 404 handler figures out the request was for "/photo/123/axbcgsfdt.jpg" and just grabs it from the database, caches it and redirects back to it. That way I get the convenience of Tcl code and the speed of static files - save for the first request. The 404 handler approach is very clever, that'll do exactly what I need, thanks! Dossy: not sure if it's even worth benchmarking this, as this approach will yield the static-file-speeds, which are amazing, except in the rare no-static-file-available case, where it's back to 900/s. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] caching tcl pages with tcl access to mark the cache as dirty
Does a module exist that does this: - register a Tcl function for a given URL, ie /photo/* - and specifying an ns_cache for the resulting rendered page - so that C code completely handles successive requests for that URL - and so that if the back-end data changes, the ns_cache can be marked as "dirty" by another tcl function (ie, if the user uploads a new photo) If not, I'm happy to write it, it's not hard, but I thought I'd ask. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks
Hmm--I'm not sure what you're thinking of or referring to, but the common "optimization" is to use sendfile(2), which seems to be alive and well in the 2.6 tree. Whoops, you're right, I just remembered a sysadmin email about sendfile not existing on the kernel we're using. Hmm... Sendfile requires that the data to be written come from a file descriptor. The assumption is that the program opens a file, then wants to pass it off to have it sent out another file descriptor, typically a network connection. But, what if you want to avoid disk I/O and cache that data in memory? sendfile() isn't necessarily workable, there. Perhaps you could mmap() to a fd and use that--should test the performance of such an implementation. I've done exactly that in the past, and found virtually no speed improvement, because Unix has a very aggressive disk cache, so adding your own cache on top of that just skips a few system calls. So, I don't see any point in worrying about improving aolserver's plain-file-sending, at least until we get start getting 10 gigabit network cards (!) I am concerned about your "hello world" dynamic request benchmark, though. I would have expected at least 4k req/sec--not the sub-1k req/sec you saw. I have a feeling it has to do with the default ns_pools/ns_limits settings, which naturally are NOT tuned for a 8- core CPU box. I bet with just a few minutes of tweaking and tuning, we can get between 4k-8k simple dynamic req/sec out of your hardware. I have an 8 cpu Mac I, but it's not on the net (it's at my home), so I can't give you ssh access. If you can suggest a thing or two to try, I'll give it a whirl. I benchmarked the "/helloworld" C module, and got about 16k responses per second vs 900/s with the tcl equivalent. I've been looking at C-caching of Tcl dynamic content, with "dirty cache" support. For example, replacing the Tcl code that returns a user's uploaded photo with C code. I wrote C code to do this, and got 14k/second vs 240/s for the same tcl function. So, C really is a good replacement for Tcl when a particular URL needs to be very fast. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks
www64:/b# ab -c 5 -n 5 http://images.bookmooch.com/x.txt Since your machine is a 2-CPU 8-core box, could you try runs with -c 8 and -c 16? It may have no effect, but it'd be nice to know that for sure. I get more-or-less the same numbers at -c 8, -c 16 and -c 32, varying about 100 requests/second between runs. I'm not sure why the tcl "hello world" test yields speeds so much slower, with ab -c 8 -n 5000 http://bookmooch.com/test.adp yielding: Requests per second:971.45 [#/sec] (mean) On the positive side, a real world function, namely a registered tcl proc for returning a member's uploaded photo from the database: ab -c 8 -n 5000 http://bookmooch.com/photo/johnbuckman.jpg (I know it looks like an image url, but it's actually tcl) yields: Requests per second:892.18 [#/sec] (mean) which is a very good real-world speed (fyi, the jpg is 6831 bytes long). -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] aolserver vs lighthttpd, benchmarks
I did some benchmarks of aolserver vs lighthttpd on plain files of various sizes, on my 8cpu 64 bit server. For 3 runs, with a 4k text file, the lighthttpd stats were 15103.87/ s, 14845.20/s and 15307.17/s, vs (as Dossy reports http://dossy.org/ archives/000517.html) aolserver's 15237.00/s. Result: Aolserver is performing virtually identically to lighthttpd with a small 4k file. With a 44k JPG (the BookMooch home page illustration) aolserver is slightly slower: aolserver: 8164.54/s, 8283.93/s lighthttpd: 10281.01/s, 9969.91/s With 1,414k sized zip, aolserver is about 1/2 as fast: aolserver: 406.36/s, 421.10/s lighthttpd: 844.05/s, 807.83/s with a 128mb file, aolserver is 40% slower: aolserver: 5.42/s lighthttpd: 8.88/s And just FYI "hello world" in tcl (<% ns_adp_puts "hello" %>): aolserver: 950.85/s I haven't done a "hello world" in C: I assume it'd be close to the 16k/s speed -john www64:/b# ab -c 5 -n 5 http://images.bookmooch.com/x.txt This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright 2006 The Apache Software Foundation, http://www.apache.org/ Benchmarking images.bookmooch.com (be patient) Completed 5000 requests Completed 1 requests Completed 15000 requests Completed 2 requests Completed 25000 requests Completed 3 requests Completed 35000 requests Completed 4 requests Completed 45000 requests Finished 5 requests Server Software:lighttpd/1.4.18 Server Hostname:images.bookmooch.com Server Port:80 Document Path: /x.txt Document Length:4070 bytes Concurrency Level: 5 Time taken for tests: 3.266443 seconds Complete requests: 5 Failed requests:0 Write errors: 0 Total transferred: 21895 bytes HTML transferred: 20350 bytes Requests per second:15307.17 [#/sec] (mean) Time per request: 0.327 [ms] (mean) Time per request: 0.065 [ms] (mean, across all concurrent requests) Transfer rate: 65458.97 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 0.0 0 0 Processing: 00 0.1 0 12 Waiting:00 0.1 0 12 Total: 00 0.1 0 12 Percentage of the requests served within a certain time (ms) 50% 0 66% 0 75% 0 80% 0 90% 0 95% 0 98% 0 99% 0 100% 12 (longest request) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver on 64-bit Linux
I'm not sure if my post is off-topic, but when I did compile Aolserver in my x86-64 machine I had problems with TCL, tDOM (not Aolserver). You can see my problem in: http://openacs.org/forums/message-view? message_id=369867 It was a problem I had installing Aolserver. I think it doesn't matter here but perhaps it is. I use tDOM on the 64 bit machine, and instead of installign tDOM from a tea file, I installed from the tarfile, with the command: ../configure --enable-threads --with-tcl=/usr/local/lib/ --enable-64bit and didn't have any problems with tDOM under 64 bit aolserver. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 64bit crashes right away
On Sep 12, 2007, at 9:47 PM, Tom Jackson wrote: Below is my configure and lib information. On Wednesday 12 September 2007 07:42, Dossy Shiobara wrote: On 2007.09.12, John Buckman <[EMAIL PROTECTED]> wrote: [EMAIL PROTECTED]:~# /lib/libc.so.6 What about /lib64/libc.so.6? And, what's "ldd nsd" say you're actually linked to? Here is mine: The nsconfig.tcl built one: [EMAIL PROTECTED]:/usr/local/aolserver.bak# bin/nsd -h Segmentation fault [EMAIL PROTECTED]:/usr/local/aolserver.bak# ldd bin/nsd libnsd.so => /usr/local/aolserver/lib/libnsd.so (0x2b2ccebf9000) libnsthread.so => /usr/local/aolserver/lib/libnsthread.so (0x2b2ccee81000) libtcl8.4.so => /usr/local/aolserver/lib/libtcl8.4.so (0x2b2ccf08c000) libdl.so.2 => /lib/libdl.so.2 (0x2b2ccf354000) libpthread.so.0 => /lib/libpthread.so.0 (0x2b2ccf559000) libm.so.6 => /lib/libm.so.6 (0x2b2ccf774000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2b2ccf9f6000) libc.so.6 => /lib/libc.so.6 (0x2b2ccfc05000) /lib64/ld-linux-x86-64.so.2 (0x2b2cce9dc000) [EMAIL PROTECTED]:/usr/local/aolserver.bak# and the working-but-sticks-on-large-pages from Tom's ./configure hack: [EMAIL PROTECTED]:/usr/local/aolserver# ldd bin/nsd libnsd.so => /usr/local/aol3/lib/libnsd.so (0x2b8ce3b39000) libnsthread.so => /usr/local/aol3/lib/libnsthread.so (0x2b8ce3dc1000) libtcl8.4.so => /usr/local/aol3/lib/libtcl8.4.so (0x2b8ce3fcc000) libdl.so.2 => /lib/libdl.so.2 (0x2b8ce4294000) libpthread.so.0 => /lib/libpthread.so.0 (0x2b8ce4499000) libm.so.6 => /lib/libm.so.6 (0x2b8ce46b4000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2b8ce4936000) libc.so.6 => /lib/libc.so.6 (0x2b8ce4b45000) /lib64/ld-linux-x86-64.so.2 (0x2b8ce391c000) appears identical to me. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 64bit crashes right away
On 2007.09.11, John Buckman <[EMAIL PROTECTED]> wrote: Is there any special trick to getting aolserver to work on 64bit linux? Which flavor of Linux? Debian? Redhat? Something else? Which version of gcc? And glibc? Ubuntu server edition 7.04 Note in my previous followup message that using the older ./configure got me no longer crashing, but now large web pages stick mid- download. Using nsconfig creates an nsd that crashes on "bin/nsd -h" Machine Details: uname -a Linux www64 2.6.20-15-generic #2 SMP Sun Apr 15 06:17:24 UTC 2007 x86_64 GNU/Linux gcc -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --enable-languages=c,c+ +,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with- system-zlib --libexecdir=/usr/lib --without-included-gettext --enable- threads=posix --enable-nls --program-suffix=-4.1 --enable- __cxa_atexit --enable-clocale=gnu --enable-libstdcxx-debug --enable- mpfr --enable-checking=release x86_64-linux-gnu Thread model: posix gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4) [EMAIL PROTECTED]:~# /lib/libc.so.6 GNU C Library stable release version 2.5, by Roland McGrath et al. Copyright (C) 2006 Free Software Foundation, Inc. Compiled by GNU CC version 4.1.2 (Ubuntu 4.1.2-0ubuntu4). Compiled on a Linux >>2.6.15.7<< system on 2007-04-04. Available extensions: crypt add-on version 2.1 by Michael Glad and others GNU Libidn by Simon Josefsson GNU libio by Per Bothner NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk Native POSIX Threads Library by Ulrich Drepper et al BIND-8.2.3-T5B Thread-local storage support included. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 64bit crashes right away
Thanks for the tip. That gets me more-or-less working, though now document over a few kb "stick". Here is what I did for a configure statement to stop crashing: ./configure --prefix=/usr/local/aol3 --enable-64bit --enable-threads --enable-shared --with-tcl=/usr/local/lib I also did: export CC="gcc -m64" and also added -nostartfiles to get around the _init redefinition failure: ns.mak:CFLAGS_OPTIMIZE = -O2 -nostartfiles that gets nsd more-or-less working. However, nsd now "sticks" on pages more than a few kb large, and a large JPG only loads 1/4 of the way. Any tips? -john On Sep 11, 2007, at 5:21 PM, Tom Jackson wrote: My laptop is 64 bit: Linux localhost 2.6.18-gentoo-r4 #1 SMP Mon Nov 20 16:49:16 UTC 2006 x86_64 AMD Turion(tm) 64 X2 Mobile Technology TL-52 AuthenticAMD GNU/Linux For Tcl I do: ./configure --prefix=/web/nsd45 \ --enable-64bit \ --enable-threads \ --enable-shared But when it builds I get the impression it isn't 64 bit. Don't know how to check it. AOLserver builds and runs as normal. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] 64bit crashes right away
Is there any special trick to getting aolserver to work on 64bit linux?
A stock nsd crashes right away (strace below).
I'm using
Linux www64 2.6.20-15-generic #2 SMP Sun Apr 15 06:17:24 UTC 2007
x86_64 GNU/Linux
The only change I made from the cvs tree was the previously mentioned
-nostartfiles mod:
CC = $(PURIFY) gcc -nostartfiles
I'm crashing running "bin/nsd -h"
-john
strace of bin/nsd -h
...
munmap(0x2b0c9308a000, 51575) = 0
open("/etc/host.conf", O_RDONLY)= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x2b0c9308a000
read(4, "# The \"order\" line is only used "..., 4096) = 92
read(4, "", 4096) = 0
close(4)= 0
munmap(0x2b0c9308a000, 4096)= 0
futex(0x2b0c93c96ca0, FUTEX_WAKE, 2147483647) = 0
open("/etc/hosts", O_RDONLY)= 4
fcntl(4, F_GETFD) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=284, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x2b0c9308a000
read(4, "127.0.0.1\tlocalhost\n#127.0.1.1\tu"..., 4096) = 284
read(4, "", 4096) = 0
close(4)= 0
munmap(0x2b0c9308a000, 4096)= 0
dup2(3, 0) = 0
write(2, "\nError: required -t opt"..., 50
Error: required -t option not specified
) = 50
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 26160 detached
and normal startup:
...
munmap(0x2b2a082a7000, 4096)= 0
open("/usr/local/aolserver/lib/libnss_files.so.2", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/lib/libnss_files.so.2", O_RDONLY) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P \0\0\0"...,
832) = 832
fstat(4, {st_mode=S_IFREG|0644, st_size=43440, ...}) = 0
mmap(NULL, 2139464, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
4, 0) = 0x2b2a082a7000
mprotect(0x2b2a082b1000, 2093056, PROT_NONE) = 0
mmap(0x2b2a084b, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|
MAP_FIXED|MAP_DENYWRITE, 4, 0x9000) = 0x2b2a084b
close(4)= 0
open("/etc/host.conf", O_RDONLY)= 4
fstat(4, {st_mode=S_IFREG|0644, st_size=92, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x2b2a084b2000
read(4, "# The \"order\" line is only used "..., 4096) = 92
read(4, "", 4096) = 0
close(4)= 0
munmap(0x2b2a084b2000, 4096)= 0
futex(0x2b2a082a2ca0, FUTEX_WAKE, 2147483647) = 0
open("/etc/hosts", O_RDONLY)= 4
fcntl(4, F_GETFD) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=283, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x2b2a084b2000
read(4, "127.0.0.1\tlocalhost\n127.0.1.1\tub"..., 4096) = 283
read(4, "", 4096) = 0
close(4)= 0
munmap(0x2b2a084b2000, 4096)= 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 26091 detached
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Switching to Trac to manage the AOLserver project?
After looking more closely at Trac [1], I'm really thinking it might be a good idea to use it to manage the AOLserver project. Does anyone have any strong feelings about this, particularly against it? I think this is a great idea. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] found a bug in nsd shutdown routine
I believe I've found a bug in queue.c which occasionally causes a
core dump at exit.
The bug is caused by this order of events at the end of NsConnThread:
poolPtr->threads.idle--;
poolPtr->threads.current--;
if (poolPtr->threads.current == 0) {
Ns_CondBroadcast(&poolPtr->cond);
}
Ns_ThreadExit(dataPtr);
the Ns_ThreadExit doesn't always get to complete, because poolPtr-
>threads.current can be set to zero before Ns_ThreadExit concludes,
and thus nsd exits (since the server is waiting threads.current to
reach zero) while the thread exit code is still running.
I believe the fix to this is to move the thread exit call
(Ns_ThreadExit) to immediately *before* the decrementing of the
current threads count, ie like so:
Ns_ThreadExit(dataPtr);
poolPtr->threads.idle--;
poolPtr->threads.current--;
if (poolPtr->threads.current == 0) {
Ns_CondBroadcast(&poolPtr->cond);
}
in my stress tests, this led to a successful nsd shutdown, even while
the server was being pounded by http requests, whereas before I
regularly crashed.
I found this bug by adding these log statements around the thread exit:
Ns_Log(Notice, "starting thread exit");
Ns_ThreadExit(dataPtr);
Ns_Log(Notice, "ending thread exit");
if you do this, and try to exit nsd during a stress test, you will
find that several connection threads display the starting message,
but never display the ending message. Sometimes this causes a crash,
sometimes not, but it's always bad form.
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Speaking of nsdci...
On Aug 9, 2007, at 2:07 AM, Nathan Folkman wrote: A variant is used in production, although it's not exactly the same. Can you say more about variant? There's only a couple of us working on updating this code at the moment, but I'd be more then happy to add anyone to the project who is interested in contributing. There are a bunch of sample configuration files, which honestly is the most confusing part of this code base. Contact me off list if interested in pitching in, and I'll get you set up. Ok, so it's messy right now, so probably not good for merging in. Anyhow, I'm not that concerned about this feature right now, but Dossy mentioned wanting to know why I used lighthttpd for large files instead of aolserver, and that's how this async issue came up. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] async background delivery
as far as what lighthttpd and mathopd are doing to get better speeds, is that they both are not multithreaded, they are just a single async loop, serving static files. I remember that this was an option in Aolserver v2, but I believe it went away in v3. Gustaf Neumann of WU-Wien patched AOLserver to do asynchronous background delivery, and has been using the feature heavily since 2005: http://openacs.org/xowiki/weblog-portlet?ptag=asynchronous http://openacs.org/forums/message-view?message_id=482221 Very nice. The only thing I'd change would be to have this not require Tcl, but rather to have registered file types that are automatically sent via the async method (ie mp3/zip) So, what's involved with having the patch applied to cvs? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Or as an alternate answer: use apache itself as the proxy. The poor saps who subject themselves to PHP will be happy and the OACS users can have a real system to work with. Are there any caching proxy plugins for aolserver? I have cheap bandwidth in other countries, which I'd like to load balance media file serving to. I've used squid in the past, but its algorithm is impenetrable, and I didn't see the bandwidth savings I thought I should. Something like this: a) receive http request b) is file available locally? b1) YES: return the file b2) NO: return HTTP redirect, and get the file from the source, so it's available for the next request c) are we out of disk space? if so, delete the least used file(s) Obviously, a very easy algorithm to write, and I was thinking of using aolserver to do this -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
On 2007.08.07, John Buckman <[EMAIL PROTECTED]> wrote:
I use AOLserver + lighthttpd on the same machine, different IPs, so
that mp3s and GIF/JPEGs are fed off lighthttpd, which is stupid and
very fast (it's wikipedia's media server)
Out of curiousity--have you benchmarked lighttpd vs. AOLserver (since
you already have both set up)? AOL replaced their "ArtBlaster" (very
thin, lightweight, single-threaded HTTP server for static assets) a
while back and replaced it with AOLserver 4.0, since it was "fast
enough" ... if lighttpd's benchmarks are significantly better, I'd
like
to try and understand why/how and start tuning AOLserver to match.
I'd just like to know if folks are running lighttpd because it
truly is
faster, or simply because it isn't AOLserver ...
Yes, two yeasr ago I did 3 benchmarks, with CGI, small GIF image, and
large mp3. I also compared aolserver to a simple all-tcl single
threaded web server which I obtained off the tcl wiki. All benchmarks
below were on a powerpc mac mini, two years ago, with "ab" and 10
simultaneous queries.
900 byte image fetch benchmark:
---
apache img fetch: 593 /s
aolserver img fetch: between 1019 and 1267/s
lighthttp img fetch: 1089/s
mathopd img fetch: 900/s
tclhttpd: 69/s
trivial http w/image cache: 1127/s
This big surprises were how well aolserver did on small images, as
well as the all-tcl web server.
requests per second handled with trivial tcl dynamic web page ("hello
world"):
---
lighttpd-cgi: 15/second
tclhttpd: 32/s
apache/php: 120/s
aolserver: between 640/s and 750/s
trivial all-tcl-web-server http://wiki.tcl.tk/15244: 1162/s
Unfortunately, I don't have any notes on the large image benchmark,
but let me put some numbers in from memory:
1mb image fetch benchmark:
---
apache img fetch: 33 /s
aolserver img fetch: around 400/s
mathopd img fetch: 750/s
lighthttp img fetch: 900/s
tclhttpd: 3/s
--
as far as what lighthttpd and mathopd are doing to get better speeds,
is that they both are not multithreaded, they are just a single async
loop, serving static files. I remember that this was an option in
Aolserver v2, but I believe it went away in v3. The async sending
feature really only kicks the performance up with large files that
require many writes to a socket. For small files that fit inside 4k,
aolserver is really fast.
What my benchmarks found is that except for possibly java-based web
servers, nothing comes close to aolserver's performance for web-
application building. For small images, aolserver is competitive
with lighthttpd. It's only with large files that the async-event-
loop approach yields better results.
One other note, is that mathopd/lighthttp use less than 5% of the CPU
running the large image benchmark. Mathopd also had an option to use
SendFile() (which is no longer available on linux) which offloaded
most of the work to the kernel.
I'm including my trivial all-tcl http server, just for reference.
What surprised me was how badly performing tclhttpd was in all
benchmarks, vs this all-tcl server design.
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
ht.tcl
Description: Binary data
Re: [AOLSERVER] aolserver focus
Idle curiosity - I wonder if anyone is running a system with both apache and aolserver listening on port 80 on different ifs/ips. Should be possible and not even difficult, tho probably of limited utility. Yes, I have setups like this and is the best solution to this problem. However, in many situations, multiple ip-addresses are unavailable and Apache and AOLserver compete for port 80. I use AOLserver + lighthttpd on the same machine, different IPs, so that mp3s and GIF/JPEGs are fed off lighthttpd, which is stupid and very fast (it's wikipedia's media server) -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] two OSX bugs
I banged on this thread-time-crash-on-OSX bug a bit, and found that setting ns_param threadtimeout 1 ;# Idle threads die at this rate gets rid of this problem. While setting ns_param threadtimeout 0 ;# Idle threads die at this rate or ns_param threadtimeout 1 ;# Idle threads die at this rate causes the problem to come back. So, it seems the problem is generally with idle threads being shut down. -john Bug #2: occasional crash on OSX, when threads give notice of timeout waiting for connection The gdb stack trace is a bit more useful in this case: [07/Aug/2007:00:05:15][16497.25228800][-nssock:driver-] Notice: nssock: listening on 127.0.0.1:81 [07/Aug/2007:00:05:24][16497.25229824][-conn:0-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25203712][-conn:5-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25230848][-conn:2-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25229824][-conn:1-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25202688][-conn:4-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25201664][-conn:3-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25203712][-conn:6-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25229824][-conn:7-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25202688][-conn:9-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25204736][-conn:8-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25201664][-conn:10-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25203712][-conn:11-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25202688][-conn:12-] Notice: exiting: timeout waiting for connection Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x0004 [Switching to process 16497 thread 0x3a0b] 0x0a00c130 in Tcl_AsyncDelete () (gdb) where #0 0x0a00c130 in Tcl_AsyncDelete () #1 0x00655500 in SignalCmdCleanUp () #2 0x0a00f23a in DeleteInterpProc () #3 0x0a00c9b1 in Tcl_DeleteInterp () #4 0x0007da21 in DeleteData () #5 0x793a in NsCleanupTls () #6 0x891f in FreeThread () #7 0x90025400 in _pthread_tsd_cleanup () #8 0x90024f88 in pthread_exit () #9 0x8203 in Ns_ThreadExit () #10 0x0006f756 in NsConnThread () #11 0x88ee in ThreadMain () #12 0x90024227 in _pthread_body () (gdb) This doesn't consistently occur, but it's often enough that I run nsd in a re-starting shell script while running on OSX. -- I wish I could be more helpful and actually give a C code patch for these, but they're not aggravating enough for me to spend the ramp- up time figuring how to fix this sort of thing, but I thought at least I should report them so they could be logged. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] two OSX bugs
I've had these two bugs for a while on OSX, thought I'd report them. I'm running today's current CVS. -- Bug #1: nszlib causes "abort trap" on OSX. 007:23:52:46][16387.2684407808][-main-] Notice: modload: loading '/ usr/local/aolserver/bin/nszlib.so' [06/Aug/2007:23:52:48][16387.2684407808][-main-] Notice: conf: [ns/ server/BookMooch]enabletclpages = 0 [06/Aug/2007:23:52:48][16387.2684407808][-main-] Notice: default thread pool: minthreads 0 maxthreads 10 idle 0 current 0 maxconns 0 queued 0 timeout 0 alloc: invalid block: 0x27e000: ef ef 80 ./aol: line 4: 16387 Abort trap sudo bin/nsd -ft config.tcl -u john -b 127.0.0.1:81 if I don't use nszlib, I have no such problems. Also, nszlib works fine for me on Linux. I haven't reported the problem before, because I develop on OSX, but my servers are Linux, so not having zlib during debugging isn't a big deal. Program received signal SIGABRT, Aborted. 0x9003d66c in kill () (gdb) where #0 0x9003d66c in kill () #1 0x9010e8cf in raise () #2 0x9010d422 in abort () #3 0x0a05788e in Tcl_PanicVA () #4 0x0a0578a9 in Tcl_Panic () #5 0x0a064fdb in TclpFree () #6 0x0a00e9c2 in Tcl_DeleteCommandFromToken () #7 0x0a0526fc in TclTeardownNamespace () #8 0x0a00f172 in DeleteInterpProc () #9 0x0a00c9b1 in Tcl_DeleteInterp () #10 0x0007d60c in PushInterp () #11 0x0007467b in NsInitServer () #12 0x0006d85e in Ns_Main () #13 0x2a7e in main () -- Bug #2: occasional crash on OSX, when threads give notice of timeout waiting for connection The gdb stack trace is a bit more useful in this case: [07/Aug/2007:00:05:15][16497.25228800][-nssock:driver-] Notice: nssock: listening on 127.0.0.1:81 [07/Aug/2007:00:05:24][16497.25229824][-conn:0-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25203712][-conn:5-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25230848][-conn:2-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25229824][-conn:1-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25202688][-conn:4-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:24][16497.25201664][-conn:3-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25203712][-conn:6-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25229824][-conn:7-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25202688][-conn:9-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25204736][-conn:8-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25201664][-conn:10-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25203712][-conn:11-] Notice: exiting: timeout waiting for connection [07/Aug/2007:00:05:25][16497.25202688][-conn:12-] Notice: exiting: timeout waiting for connection Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x0004 [Switching to process 16497 thread 0x3a0b] 0x0a00c130 in Tcl_AsyncDelete () (gdb) where #0 0x0a00c130 in Tcl_AsyncDelete () #1 0x00655500 in SignalCmdCleanUp () #2 0x0a00f23a in DeleteInterpProc () #3 0x0a00c9b1 in Tcl_DeleteInterp () #4 0x0007da21 in DeleteData () #5 0x793a in NsCleanupTls () #6 0x891f in FreeThread () #7 0x90025400 in _pthread_tsd_cleanup () #8 0x90024f88 in pthread_exit () #9 0x8203 in Ns_ThreadExit () #10 0x0006f756 in NsConnThread () #11 0x88ee in ThreadMain () #12 0x90024227 in _pthread_body () (gdb) This doesn't consistently occur, but it's often enough that I run nsd in a re-starting shell script while running on OSX. -- I wish I could be more helpful and actually give a C code patch for these, but they're not aggravating enough for me to spend the ramp-up time figuring how to fix this sort of thing, but I thought at least I should report them so they could be logged. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
On Aug 3, 2007, at 8:41 PM, Rick Cobb wrote: John, does this distro's Http package work well within AOLServer? We did a cURL integration for AOLServer 3.4.2, but have been holding off on any contribution because it's very intertwined with our C++ stuff -- and figured that one reason the AOLServer community did all its changes for 4.0 & 4.5 was to be able to use packages like that. We haven't gotten far enough in our port to 4.5 to have tested the Http package (we've been focused on C/C++ integration, not Tcl, so far). Yes, the http package works perfectly inside AOLServer, I use it for Amazon API XML lookups on BookMooch, which is a heavily used feature, so definitely no problems. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
On new aolserver installations, I install the ActiveState "batteries included" tcl version, and then copy over all the libraries it has (which is a *lot*) into aolserver's tcl directory (in my case /usr/local/), which makes for an extremely capable AolServer/tcl distro. hmm.. it might actually be possible to build aolserver against the activestate distro directly to accomplish this. The ActiveState "batteries included" tcl version competes really well, IMHO, with PHP and Perl. My only issue has been that apparently TclX doesn't play nice with AOLserver and can cause unclean shutdowns (I think Dossy said this), otherwise I have a wide variety of libraries, pretty much all the same stuff as PHP/ Perl. I've said before that AOLServer would best be served as a series of modules that leverage standard distributions. In this way it would become more like Ruby on Rails or similar Python frameworks. A more elegant coupling will probably improve things all around. To that end, I can assist with the de/recoupling, although my time resources for me are unfortunately not in abundance. Would it be even possible to have AOLServer as a tcl module, and thus automatically distributed with the batteries included releases? As to the TclX problem ... I wonder if that has to do with mt issues. I don't know if TclX has been fully vetted for thread- safety (and you should most definitely never fork() with threads). Yeah, that would make sense, though it is completely stable for except during shutdown. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
I'm wondering -- does it make sense to just try to close the gap with LAMPP as a model, driving to the "batteries-included" distro Dossy's been talking about for years? That seems to me like a project tons of folks could contribute too -- from docs to extensions to installers, etc. On new aolserver installations, I install the ActiveState "batteries included" tcl version, and then copy over all the libraries it has (which is a *lot*) into aolserver's tcl directory (in my case /usr/ local/), which makes for an extremely capable AolServer/tcl distro. hmm.. it might actually be possible to build aolserver against the activestate distro directly to accomplish this. The ActiveState "batteries included" tcl version competes really well, IMHO, with PHP and Perl. My only issue has been that apparently TclX doesn't play nice with AOLserver and can cause unclean shutdowns (I think Dossy said this), otherwise I have a wide variety of libraries, pretty much all the same stuff as PHP/Perl. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] configured minthreads, maxthreads doesnt show up with [ns_server threads] command
On Aug 3, 2007, at 3:39 PM, [EMAIL PROTECTED] wrote: Please watch the vulgar language - there's simply no need for it. And precisely who are you to say so? He's someone who is trying to keep the discussion civil and productive, and trying to be helpful. Really, I think it's best to express yourself in a non- confrontational way, you're much more likely to have the other side listen to you. Folks can we tone down the flamage on this list? We're a small community, and nobody will be served well by angering each other. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] learning from naviserver
So, in the spirit of open source software meritocracies: please place your money where your mouth is. Come up with a list of actionable changes you'd make if you were king. Lets hear it--and if everyone agrees to a particular change, we'll declare it made. (Note: declaring anything to a volunteer-driven organization doesn't guarantee that anyone will actually do it.) In no way do I want to be king, and in an effort to calm things down a bit, let me say that I'm really, really happy with aolserver, and the last major release had amazing things in it. My only real nit is that so many cool things are not-so-well-documented, but hell, it's open source. I just spot things in the text files, and then figure it out by reading the C code, which is often commented, but always clean and readable. On the subject of cool things and not-so-well-documented, I'd like to bring up Naviserver (you can find it on sourceforge, it's an independent fork of aolsever, at http://naviserver.sourceforge.net). Their fork of aolserver has an insane number of changes to it, and lots of great ideas (look at http://naviserver.cvs.sf.net/naviserver/ naviserver/ChangeLog?view=markup). There's a handful of developers working on it, and it seems like a real hotbed of innovation. I've not used naviserver, though I evaluated it seriously, because it has so many differences to aolserver, almost all undocumented, that it was really hard for me to get up to speed to, and I found it less reliable than aolserver, probably because of all those innovations. I kind of like the slower pace of aolserver, I can actually run a production web site on it. This is not meant as an insult to Vlad and the other Naviserver developers, I'm just pointing out how the two development communities differ. However, I was wondering if we should perhaps look at merging back some of the best things in naviserver, back into aolserver. In fact, maybe we should treat the aolserver/naviserver split like ubuntu treats its two releases, and recognize that naviserver as an innovative, highly chaotic playground, and merging back in the best ideas from it back into aolserver after a long delay (6 months to a year) once each feature has settled down a bit and we can evaluate whether, in hindsight, it really was a good idea and the way it was implemented turned out well. I'd be game to go through the naviserver changelog in the future, and be part of a discussion of what is in there that we might want to merge back into aolserver. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_zlib compress
How do you enable gzip page compression on ns_register_proc pages?
The adp gzip compression occurs in C code, in Ns_ConnFlush(), which
checks for "Accept-Encoding: gzip"
Since a "ns_register_proc" uses "ns_return", like this:
ns_return $conn 200 text/html $html
I believe it's bypassing this gzip code, and always returning html.
I wrote a ns_return_gzipped() proc (see below), but it doesn't seem
to work, so I was wondering if anyone's done this already, and how.
FYI I switched BookMooch to use adp gzip compression, and the results
are impressive. 200k html files go to 35k, greatly speeding up the
user experience.
-john
proc ns_return_gzipped {conn html} {
if {[string first {gzip} [ns_set get [ns_conn headers] {Accept-
Encoding}]] != -1} {
set zl [ns_zlib gzip $html]
ns_set put [ns_conn outputheaders] Content-Encoding gzip
ns_return $conn 200 text/html $zl
} else {
ns_return $conn 200 text/html $html
}
}
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_zlib compress
I have ns_param gzip on , but do i still need to set some headers ? I don't know. How do i know its gzipping anything ? telnet www.yourwebserver.com 80 And send a GET request. That won't invoke compression unless you include the appropriate Accept-encoding header. You need to sent a complete request like If you’d like to see whether a page is compressed, use Firefox and install the “Live Headers” add-on from http:// livehttpheaders.mozdev.org/. With this add-on, when you choose “info” on a page, a new tab titled “headers” is now available. Click on that tab, and look for “Content- Encoding: gzip” on the bottom window (i.e. the “Response Headers”). If there’s no “Content-Encoding” then gzip is not being used on that page. You’ll also see the transferred size of the file (2022 bytes in this case) and if you save the file to disk, you can see what the real size is (12k in this case) and thus what the speedup was. from: http://blog.bookmooch.com/2007/04/20/gzip-compression-of- bookmooch-pages-faster-faster/ -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_zlib compress
Below is a ns_return_gzipped proc that replaces ns_return and sends
back gzipped output.
It worked fine everywhere, *except* with a debugging http proxy I
used (in ActiveState's Komodo) where pages stalled. I was thinking
perhaps the problem is that "Connection: close" wasn't being honored
by the proxy, but it makes me nervous, because the builtin ADP gzip
compression works fine with the proxy.
Is there a way to force a connection closed in a ns_register_proc
page, after ns_write? I can't use "ns_conn close" in this context,
since that's an ADP command.
-john
proc ns_return_gzipped {conn html} {
set zl [ns_zlib gzip $html]
set h [subst {HTTP/1.0 200 OK
MIME-Version: 1.0
Server: AolServer/4.5.0a
Connection: close
Content-Type: text/html; charset=iso-8859-1
Content-Length: [string length $zl]
Content-Encoding: gzip
$zl}]
ns_write $conn $h
return TCL_OK
}
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_zlib compress
On Apr 19, 2007, at 10:11 AM, Rusty Brooks wrote:
I don't believe you can use ns_return with binary data.
By the way, are there any plans to change that? Or make a
ns_return_binary or something? I have a lot of code that writes to
a file and then does ns_returnfile
ns_write is what you should use for binary data:
set h [subst {HTTP/1.1 200 OK
MIME-Version: 1.0
Server: MoochServer/4.5.0a
Content-Type: image/jpeg
Content-Length: [string length $d]
Connection: close
$d}]
ns_write $conn $h
return TCL_OK
so I guess I should do that for my ns_compress code, but anyhow,
that's not the problem
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] profiling an aolserver site
I added server-side logging of ADP page execution a while back. Maybe that would help you? ns_section "ns/server/server1/module/nslog" ns_param logreqtime true Yes, that's great. Will it also log ns_register_proc pages? Have you written anything to crunch the numbers? If not, no problem, I can do that. -john [MacBook-Pro:~] nfolkman% tail -f /usr/local/aolserver/servers/ server1/modules/nslog/access.log 192.168.1.129 - - [11/Apr/2007:10:46:28 -0400] "GET /flash/ HTTP/ 1.1" 200 191 "" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv: 1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0.106342 192.168.1.129 - - [11/Apr/2007:10:46:28 -0400] "GET /favicon.ico HTTP/1.1" 404 534 "" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en- US; rv: 1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0.000361 On 4/11/07, John Buckman < [EMAIL PROTECTED]> wrote: I'm wondering if anyone on this list has written code to "profile" a web site running under Aolserver. By this, I mean, timing the start/stop time of every page, logging it, and then running a bit of analysis to find out what pages are the slowest running and which pages are the most commonly loaded, then multiplying the two (ie execution time x requests per day=total machine load per page per day) In traditional systems programming, profiling is a common tool used to determine what code should be optimized. I'd like to do the same inside aolserver. One efficient alternative I was thinking about would be to patch ns_log to include both the start request time, and the time the page was returned, in the log. That could be done if ns_log is called after the page is rendered, and I don't know if that's the case. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to < [EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- Nathan Folkman [EMAIL PROTECTED] -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] profiling an aolserver site
On Apr 11, 2007, at 9:00 AM, patrick o'leary wrote: Something that might be of use is the tcl-lib profiler http://www.tcl.tk/community/tcl2004/Tcl2003papers/kupries-doctools/ tcllib.doc/profiler/profiler.html Which gives a little insight into proc calls. Might be of use to you, obviously use with care and don't enable it on all production requests. The tcllib profiler doesn't like living inside aolserver, because it redefines "proc" and some other core things and aolserver doesn't like that. I've tried to use it in the past, but not with success. Instead, I make sure the adp page calls a tcl proc, and then I profile the tcl proc on the tclsh command line, which works ok. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] profiling an aolserver site
I'm wondering if anyone on this list has written code to "profile" a web site running under Aolserver. By this, I mean, timing the start/stop time of every page, logging it, and then running a bit of analysis to find out what pages are the slowest running and which pages are the most commonly loaded, then multiplying the two (ie execution time x requests per day=total machine load per page per day) In traditional systems programming, profiling is a common tool used to determine what code should be optimized. I'd like to do the same inside aolserver. One efficient alternative I was thinking about would be to patch ns_log to include both the start request time, and the time the page was returned, in the log. That could be done if ns_log is called after the page is rendered, and I don't know if that's the case. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] easy way to tell all tcl interps to exit next time?
On Apr 4, 2007, at 3:01 PM, Rusty Brooks wrote: would ns_eval [ns_markfordelete] work? ns_eval runs a bit of tcl code in each thread. See, I knew there ought to be an easy way! Thanks! Thanks Dossy & Rusty! Personally, I don't use packages/package require, I have an init script that loads everything. Additionally, all my pieces of code are broken up into modules. So I can say something like "module_load /module1" and it'll run the init for that module, updating all the existing threads. If I say "module_load /" then it'll load all the modules (they are hierarchically organized) P.S. the first and only time I've used packages in aolserver was a port of Listmanager, which you're obviously familiar with. When I want to reload stuff I actually do a ns_eval myinit where myinit just reloads everything in custom and htdocs/libtml. It's crude but it works for my purposes. I kind of do the same thing, in that I have an init proc that does a "package require" of everything my aolserver app needs, and if rerun it first does a "package forget" on all of them. While in development, every page load causes a "package forget" so that I reload everything, so that every code change becomes immediately apparent. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] easy way to tell all tcl interps to exit next time?
Can someone suggest an easy way to tell aolserver to reload its tcl interps at the next opportunity? What I basically need is a way to run ns_markfordelete for every tcl interpreter nsd has open. http://panoptic.com/wiki/aolserver/Ns_markfordelete When I push out a new version of BookMooch (via cvs) to the live server, I currently restart nsd on the production server. What I'd like to do instead, is to have all my interps exit at their next opportunity, so that they use the new tcl libraries that are loaded with "package require". I can think of a few ways to do it, but they're all somewhat complicated, so I was wondering if anyone had an easy way to do it? ie, I could use "nsv_set exit_interp_now 1" that each interp would check at the end of each page render, but with many tcl interps in separate threads, it's tricky to make sure each interp exits exactly one time (ie, how to tell whether an interp that has exited once to not exit again when it reloads). -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] unclean shutdown
On Mar 17, 2007, at 5:24 AM, Dossy Shiobara wrote: On 2007.03.16, John Buckman <[EMAIL PROTECTED]> wrote: package require Tclx Good, that confirms my guess. From your previous stacktrace: (gdb) where #0 0x0a00c254 in Tcl_AsyncDelete () #1 0x00649500 in SignalCmdCleanUp () #2 0x0a00f35e in DeleteInterpProc () ... That crash is happening in Tclx's thread cleanup code: http://tclx.cvs.sourceforge.net/tclx/tclx/generic/tclXsignal.c I'm surprised this doesn't cause instability at runtime, but I'm guessing you don't routinely send the nsd signals. Right, I don't use the nsd signals. Is there a patch for that TclX problem? There hasn't been a TclX update in 16 months (http:// sourceforge.net/project/showfiles.php?group_id=13247) but I haven't checked their CVS tree. When you shut down, are you doing it by issuing a "ns_shutdown" via the control port (nscp), or by sending it a signal? ns_shutdown -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] unclean shutdown
On Mar 16, 2007, at 2:50 PM, Dossy Shiobara wrote: On 2007.03.16, John Buckman <[EMAIL PROTECTED]> wrote: What patchlevel of Tcl? 8.4.14 - current release. What modules are you loading? Are you using any of the Tcl async. stuff (introduced in 4.5) in your application code? I don't think I'm using any async stuff, at least not in any code I've written, hopefully no library uses it automatically. Here's a list of modules I'm using. nsd is totally stable, running for weeks w/o a reboot, it's just at shutdown that I have a problem, and I should mention that it's not every time. About 75% of the time I do get a clean shutdown. package require ncgi package require struct::list package require tdom package require csv package require crc32 1.3 package require http package require html 1.3 package require Tclx -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] unclean shutdown
On Mar 16, 2007, at 6:13 AM, Dossy Shiobara wrote: On 2007.03.15, John Buckman <[EMAIL PROTECTED]> wrote: Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x0004 [Switching to process 4464 thread 0x4613] 0x0a00c254 in Tcl_AsyncDelete () (gdb) where #0 0x0a00c254 in Tcl_AsyncDelete () #1 0x00649500 in SignalCmdCleanUp () #2 0x0a00f35e in DeleteInterpProc () #3 0x0a00cad5 in Tcl_DeleteInterp () ... What patchlevel of Tcl? 8.4.14 - current release. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] unclean shutdown
My shutdowns of aolserver are rarely clean, nsd usually crashes on shutdown. I've started running nsd inside gdb, so I can see where the crash is. Here is a stack trace on a ctrl-c, using the latest cvs source. If there's anything else I can do to help track down these unclean shutdowns, let me know. Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x0004 [Switching to process 4464 thread 0x4613] 0x0a00c254 in Tcl_AsyncDelete () (gdb) where #0 0x0a00c254 in Tcl_AsyncDelete () #1 0x00649500 in SignalCmdCleanUp () #2 0x0a00f35e in DeleteInterpProc () #3 0x0a00cad5 in Tcl_DeleteInterp () #4 0x0007da11 in DeleteData () #5 0x793a in NsCleanupTls () #6 0x891f in FreeThread () #7 0x90024f60 in _pthread_tsd_cleanup () #8 0x90024ae8 in pthread_exit () #9 0x8203 in Ns_ThreadExit () #10 0x0006f6b6 in NsConnThread () #11 0x88ee in ThreadMain () #12 0x90023d87 in _pthread_body () -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] tdom leak fix
FYI, on the tdom leak, I found that my app (BookMooch) was leaking as well (thanks for mentioning the leak problem you're having, it helped me find mine), and the reason is that tdom NEVER automatically releases memory for the documents it creates (the docs say " the document object command has to be explicitly deleted "). Therefore, I highly recommend adding a $doc delete to manually delete any tdom objects you create. Below are the relevant docs for tdom for this. -john dom parse ?options? ?data? Parses the XML information and builds up the DOM tree in memory providing a Tcl object command to this DOM document object. Example: dom parse $xml doc $doc documentElement root parses the XML in the variable xml, creates the DOM tree in memory, make a reference to the document object, visible in Tcl as a document object command, and assigns this new object name to the variable doc. When doc gets freed, the DOM tree and the associated Tcl command object (document and all node objects) are freed automatically. set document [dom parse $xml] set root [$document documentElement] parses the XML in the variable xml, creates the DOM tree in memory, make a reference to the document object, visible in Tcl as a document object command, and returns this new object name, which is then stored in document. To free the underlying DOM tree and the associative Tcl object commands (document + nodes + fragment nodes) the document object command has to be explicitly deleted by: $document delete or rename $document "" -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] mutex try lock?
So, the question becomes: is it better to add "-timeout time" to all the blocking subcommands, or to add "ns_mutex trylock"? In theory, "-timeout 0" should perform the trylock equivalent. I'd vote for -timeout on all lock requests, as that way I get both the "see if lock is available" functionality, as well as "lock it if it's available, but give up after a few seconds if it isn't" Generally, with any code I put into production, I try to put a sanity check in so that spectacular unexpected failures cause LOUD log warnings.,For normal semaphore locks, I might want to give up after 1 minute and display a warning in the log that there's likely a coding error in there (and I can even show a tcl stack trace to help find it). Otherwise, it's down to gdb and a core file to figure out what semaphores were blocking, and that's kind of ugly. I'm currently using nsv_incr to emulate this behavior, and it works well enough. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] mutex try lock?
On Feb 22, 2007, at 8:21 PM, Nathan Folkman wrote: For John. Just curious as to what problem the "try lock" was needed for. I'm trying to block the "double submit" problem on web pages, where a page submit takes too long (because something else is taking up the server), so the user hits cancel and resubmits. When the original taking-up-the-server code completes, now the two submissions occur at exactly the same moment, and the submission is processed twice. I currently use a database (check the database to make sure this operation hasn't previously run) call to prevent this, but if the two pages occur at the exact same moment, the database call isn't atomic enough to prevent the double-submit. I previously used a simple mutex to prevent the double-submit problem, but would occasionally get a deadlock when deployed in the field. The deadlock is probably due to some operations needing two locks, and thus locking against each other, and is debuggable, but with some work and with quite negative consequences on a live server, since I have to shut down the server to unlock the jammed threads. Instead, to solve the "double submit" problem, I planned a more simple strategy: 1) try to grab a mutex, so I'm the only one doing this 2) if I can't grab a mutex, wait 1 second, and try again 3) after 5 seconds, give up on the mutex, and proceed anyway, since if there is a double-submit, chances are the two threads aren't running at the same moment any more. And, display an error, so that the programmer can investigate and see if there is a deadlock-causing bug in this code. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] mutex try lock?
On Feb 23, 2007, at 8:31 AM, John Buckman wrote:
Should also get around to implementing "ns_mutex eval $mutex
$script"--acquire the mutex, evaluate the script, release the mutex.
Catch any Tcl errors and re-throw them. This guarantees that the
mutex
is unlocked, which avoids the trivial deadlock case:
mutex eval would be very handy, and "trylock" is fine by me, though
"tryandlock" would be handy, where the lock is obtained if it can be,
and if it can't be the function returns immediately with an error.
The C code for mutex lock appears to already support this (an E_BUSY
flat, if memory serves) but I didn' see a way to set a pthread to
return failure if a lock isn't possible.
To get around the error-causes-mutex-to-not-be-unlocked problem, I
created a "lock object" that automatically unlocks when the function
that created it goes out of scope.
You create a lock like this:
proc foo {
mooch_lock aLock
stuff...
}
(mutex unlocked automatically when function exits)
and this is the lock/unlock code:
proc mooch_lock {name} {
set mutex [nsv_get . $name]
uplevel 1 "set $name \"$mutex\";trace add variable bar unset
\"mooch_unlock $name\""
puts "locking $name / $mutex"
ns_mutex lock $mutex
}
proc mooch_unlock {name args} {
set mutex [nsv_get . $name]
puts "unlocking $name / $mutex"
ns_mutex unlock $mutex
return
}
that way, you don't need to use "eval" around code that is mutex
protected. The "trace" function handle mutex unlocking for you.
None-the-less, I still occasionally get a deadlock on my site, so
either this mechanism above isn't as robust as I think it should be,
or I have double-lock depending code somewhere on my web site.
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] mutex try lock?
Forgot to mention that I know about Dossy's comment in the wiki that
nsv_incr can be used to this effect, and I'm doing that.
http://panoptic.com/wiki/aolserver/Nsv_incr
but I was wondering why "try" wasn't part of ns_mutex when the C code
is there for it.
-john
On Feb 22, 2007, at 3:09 pm, John Buckman wrote:
I'm looking to have mutexes with timeouts, and I see support in the
C code for this but none carried over to Tcl.
In the C code, there's a Ns_MutexTryLock() function, but no tcl
function for calling it.
Ns_MutexLock calls Ns_MutexTryLock() and there appears to be
timeout support:
if (!NsLockTry(mutexPtr->lock)) {
return NS_TIMEOUT;
}
but I don't see any way of setting the mutex timeout seconds.
It's trivially simple to modify NsTclMutexObjCmd (in tclthread.c)
to support "ns_mutex try" and I'm wondering if there's a reason
this hasn't been done.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
[AOLSERVER] mutex try lock?
I'm looking to have mutexes with timeouts, and I see support in the C
code for this but none carried over to Tcl.
In the C code, there's a Ns_MutexTryLock() function, but no tcl
function for calling it.
Ns_MutexLock calls Ns_MutexTryLock() and there appears to be timeout
support:
if (!NsLockTry(mutexPtr->lock)) {
return NS_TIMEOUT;
}
but I don't see any way of setting the mutex timeout seconds.
It's trivially simple to modify NsTclMutexObjCmd (in tclthread.c) to
support "ns_mutex try" and I'm wondering if there's a reason this
hasn't been done.
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] build problem with extension
In case anyone needs this info in the future, I fixed this problem below. Evidently, on osx, a dynamic library and a static library are created, and at least in my case, the default aolserver module makefiles were not listing the module's .o file in the link list for the static library, only for the dynamic library. I was able to determine the fault with the "nm" command, run on the .o and .dylib files gave me a happy list of exposed functions, while the .so was stripped. So, I modified the default build command for the module, from this: gcc -pipe -bundle -o nsdbbdb.so -L. -lnsdbbdb -lnsd -lnsthread - L/usr/local/lib -ltcl8.4 -lpthread -framework CoreFoundation to now list nsdbbdb.o in the link list. Which then gave me some link errors, solved by adding BerkeleyDB, and ns_db's libraries in the link list: -L/usr/local/aolserver/lib -L/usr/local/BerkeleyDB.4.4/lib/ -lbdb -lnsdb and then aolserver ran happily with my module, and "nm" shows all the functions correctly exposed. -john On Oct 10, 2006, at 6:30 PM, John Buckman wrote: I'm having a problem loading the berkeley db extension, built on MacOSX/Intel, that I haven't had on MacOSX/PowerPC, gcc 4.01. The reason I'm inquiring here (and not with Vlad, who doesn't know either) is it seems like a general compiler/platform issue Aolserver, on load displays this error: > Warning: modload: could not find Ns_DbDriverInit in /usr/local/ aolserver/bin/nsdbbdb.so > Error: dbdrv: failed to load driver 'berkeleydb' yet Ns_DbDriverInit is defined in the .c source code > NS_EXPORT int Ns_DbDriverInit(char *hModule, char *configPath) This looks to me like a simple problem of the function not being exported correctly. Or is it a problem of gcc in c++ vs C mode? Or some compile time flag magic that's needed... Any suggestions? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] build problem with extension
I'm having a problem loading the berkeley db extension, built on MacOSX/Intel, that I haven't had on MacOSX/PowerPC, gcc 4.01. The reason I'm inquiring here (and not with Vlad, who doesn't know either) is it seems like a general compiler/platform issue Aolserver, on load displays this error: > Warning: modload: could not find Ns_DbDriverInit in /usr/local/ aolserver/bin/nsdbbdb.so > Error: dbdrv: failed to load driver 'berkeleydb' yet Ns_DbDriverInit is defined in the .c source code > NS_EXPORT int Ns_DbDriverInit(char *hModule, char *configPath) This looks to me like a simple problem of the function not being exported correctly. Or is it a problem of gcc in c++ vs C mode? Or some compile time flag magic that's needed... Any suggestions? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] gzip output compression
Hi, you can take a look at some of the included documentation for
the new "nszlib" module:
http://aolserver.cvs.sourceforge.net/aolserver/aolserver/nszlib/
README?view=markup
http://aolserver.cvs.sourceforge.net/aolserver/aolserver/nszlib/
example.tcl?view=markup
Hmm.. I thought I had read something about the zlib compression
kicking in automatically at a larger than X size document being
returned. Or is gzip support in AS different than zlib support
(since gzip uses zlib internally, nay? http://www.gzip.org/)
I posted about this in June:
http://www.mail-archive.com/aolserver@listserv.aol.com/msg09964.html
-john
---
Re: [AOLSERVER] gzip compression
John Buckman
Mon, 26 Jun 2006 00:49:53 -0700
Having said that, the preferred future direction seems to be
ns_adp_compress:
http://sourceforge.net/tracker/index.php?
func=detail&group_id=3152&atid=353152&aid=1099613
Does this gzip option work with the AOLserver source in the current
CVS tree?
I tried it, and added the recommended config section:
> ns_section "ns/server/${servername}/adp/gzip"
> ns_param enabled on
> ns_param level 4
> ns_param minsize 0
and couldn't get a compressed result back with:
> curl -H "Accept-Encoding: gzip" http://londonmini
Which I think should work.
Digging in the source, I see references to config options "gzipmin"
and "gziplevel"
if (!Ns_ConfigGetBool(path, "gzip", &i) || i) {
servPtr->opts.flags |= SERV_GZIP;
}
if (!Ns_ConfigGetInt(path, "gzipmin", &i) || i <= 0) {
i = 4 * 1024;
}
servPtr->opts.gzipmin = i;
if (!Ns_ConfigGetInt(path, "gziplevel", &i) || i < 0 || i > 9) {
i = 4;
}
servPtr->opts.gziplevel = i;
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Tcl interpreter bloating
On Oct 2, 2006, at 11:10 PM, Nathan Folkman wrote: [EMAIL PROTECTED] wrote: This morning my server complained: "unable to alloc 2169111 bytes" How large was the nsd process when it crashed? Not sure how big, I wasn't around. But, nsd regularly grows to 2gb, and I usually shut it down around 1.5gb. I've got nsd shutting down with ns_shutdown every 24h now, and auto- restarting. That's a temporary fix that will work fine for now, and is pretty clean. It also gives me an oppty to do a database backup every 24h, which I like. Using ns_markfordelete every 100 interp passes, had no effect on stopping my nsd bloating, which makes me think it's NOT the tcl memory allocator fragmenting, because it seems to be that a tcl interp exiting would free its memory back to the process for reuse. So, next on my agenda is moving off the berkeleydb/aolserver nsdb driver, to see if that fixes the bloating. If so, then I'll worry about performance. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl interpreter bloating
Yes, I prototyped exactly that, and it worked, and yield only a 4x slowdown in database fetch time, rather than the 18x slowdown that the RPC mechanism yielded. However, I probably won't initially use that technique, as I first want to see where the bloat problem is. Uh, if you use nsproxy to push the BDB access out to a proxy pool, it'll help you isolate the problem, won't it? nsproxy pools are pools of separate processes. If you push your BDB accesses out to a nsproxy pool, but your nsd memory footprint still grows, then you know the issue is elsewhere in your code that's still running in your main nsd. Right, good catch, I forgot that nxproxy pools run as separate processes, so yes, that would help catch it. However nsd exits hard, then the nsproxy pools exit hard too, risking database corruption. Is there a way to run an nsproxy pool started by a separate process from nsd, so that if nsd crashes the nsproxy pool keeps running. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl interpreter bloating
This morning my server complained: "unable to alloc 2169111 bytes" and thus exited. So, I'm thinking that malloc fragmentation may very well be the problem. I read recently that the connsperthread parameter no longer works in aolserver. I believe this parameter destroys a Tcl interpreter and recreates it, yes?. This technique seems to me a much better way to deal with tcl memory fragmentation than experimenting with a different malloc (ie, it's parallel to the way Apache deals with memory bloat in mod_perl). I'm trying (right now on the live server) to do a manual version of connsperthread, using ns_markfordelete and a per-interpreter counter, cleaning up an interpreter after 1000 uses. I'll be curious to see if that fixes the bloat problem. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl interpreter bloating
I still have been unable to get nsproxy to work. Did you try that thing I sent you a few weeks back? (Basically nsproxy would work for a few minutes and then crash the server) FYI, nsproxy worked fine for me under macosx and linux. I believe Rusty may be running it under windows (can you confirm?). -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl interpreter bloating
FYI, Berkeley DB's Tcl binding isn't truly thread-safe: http://dossy.org/archives/000307.html I'm not using their Tcl binding, as it's not even close to thread safe. I'm using Vlad's BerkeleyDB/aolserver nsdb driver. However, that may be leaky, so what I was going to do is make a simple tcl rpc server that uses the Berkeley DB tcl binding, and have aolserver talk tcpip rpc to that. It'll be slow, but it'll show me if the bloat is the berkeley db nsdb driver or not. I'm not sure if this can be "fixed" by implementing a BDB nsdb driver for AOLserver that uses the BDB C API or not. That *is* what I'm using currently, and Vlad's driver works very well, but I do have bloat and I don't know where it comes from. It's quite bad now, going to 1.3g of memory use in 24h. What you can consider doing is using the new nsproxy module to use the BDB Tcl binding from nstclsh in separate processes in a nsproxy pool. Then, you're relying on BDB's "concurrency" (multi-process locking) being implemented correctly. Yes, I prototyped exactly that, and it worked, and yield only a 4x slowdown in database fetch time, rather than the 18x slowdown that the RPC mechanism yielded. However, I probably won't initially use that technique, as I first want to see where the bloat problem is. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl interpreter bloating
Your problem is most likely the Tcl memory allocator, and fragmentation that can occur within the pools. There have been some posts detailing these issues a few months ago I think. Ah, ok. One thing to try is building Tcl so that it doesn't use the threaded-allocator, and to instead try something like Hoard or Google's tcmalloc. Now all that aside, is it really a big deal to do rolling restarts? Right restarting is not something I like to doing, because most of the time when I ctrl-c aolserver, it exists uncleanly with one of these errors: 21597 aborted or called TclFindHashEntry on deleted table or open database file handle at exit: (db name) and side-effects of this unclean exit are: 1) incomplete database operations (and I don't use transactions, so I get dangling records) that need to get cleaned up (ie, referential integrity is broken) 2) the possibility of database corruption (since I'm using berkeleydb) if I shut down hard in the middle of a database write, which has occurred a few times, and required an export to text/ reimport to fix. 3) anxiety All these things *may* be caused by bugs in the berkeleydb/aolserver driver, I realize this. So, I'm going to a simple Tcl RPC mechanism, which will be slower (18x, by my benchmarking), but safer and then I'll know if the bloat & stability problems are due to berkeley or aolserver/tcl. Also, I'll be able to do scheduled restarts if I separate aolserver from the database server, rather than having them both be in process. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl interpreter bloating
So did setting connsperthread take care of the bloating? I keep seeing assertions that memory use by AOLserver should level off at some steady state - perhaps controlled by this parameter. But I have yet to see any one come back and say "I changed X to Y and now my server doesn't behave like it has a memory leak". FYI, BookMooch.com bloats to 2gigs within 4 days of use and needs to be restarted. I'm hoping that the problem is with the berkeleydb driver AOLserver driver, and am going to dry another database connectivity technique (RMI) to see if that fixes it. If not... -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] connected socket pool
I'd like to keep a permanently connected socket connection to another
machine, with each aolserver thread having a socket that is already
connected to that other app.
I tried using the namespace trick to keep a global around, like so:
namespace eval sbuff {}
proc mysockget {} {
global sbuff:mysock
if {[info exists mysock] == 1} {
return $mysock
}
set mysock [socket localhost ]
}
but while the socket descriptor stays around with this trick,
aolserver automatically closes the socket itself. This must be some
sort of cleanup code in aolserver, normally a good thing.
Is there a way to either:
1) have my socket NOT be cleaned up by aolserver
2) have a connected socket pool, like nsdb, but w/o any functionality
other than a connected socket
I'm not alone with this need, I know that Rusty needs to do this too,
and lots of tcpip socket protocols have login steps, that cause a new-
connection-per-adp-page strategy to be inefficient.
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] AOLserver's documentation woes and its future
On Sep 5, 2006, at 6:00 PM, Jeff Hobbs wrote: One thing I really miss in the Aolserver/Tcl world is a good debugger -- "puts" isn't such a good alternative :D. Javascript, I believe, has nice development environments and debuggers available. It'd also be nice to have a wider world of source code available to us, as Tcl has a lot, but nothing compared to the Perl/PHP world. The Tcl Dev Kit Debugger should be able to be inserted into the AOLServer environment for effective debugging. You currently couldn't do that with Komodo, but if there was demand, we could make some modifications. However, TDK can handle multiple simultaneous debugging sessions, so it should really be adept at the debugging needs of something like AOLServer. Jeff, I agree, BUT feel free to express my vote for a Mac OSX version of Tcl Dev Kit. Komodo is on OSX, but as you indicated, the Komodo Tcl debugger won't work with aolserver (I worked with your activestate tech support folks and they said it wasn't supported) As to "why a debugger for aolserver"? Any large web-based application shares many of the same complexity problems of traditional applications, and from my C++ days, I learned that I should never leave code in that I hadn't stepped through at least once. Lots of the bugs on BookMooch (22,000 lines of Tcl procs at launch) have been logic errors, and in the beta and first month of launching BookMooch, my bug database reports 414 separate bugs fixed. That's a lot of code, and a proper debugger is nice. One nice thing tclhttpd has is a debug mode that gives you a stack track, and a few other introspection features when a Tcl error occurs. Might be worth borrowing the concept. If it weren't for Komodo doing syntax checks as I type my Tcl code, I'm sure I'd have many more bugs (that's an amazing feature, folks, if you don't know about it) -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver's documentation woes and its future
Perl thread safety has never been properly debugged. In fact, that was the reason I moved from Perl to Tcl many years back. I had assumed that Python had fixed the global semaphore thing from when I looked at it 8 years ago, but no. Ok, Dossy, I buy your argument that the other pop languages aren't going to perform better because of their thread safety issues. Your server-side javascript argument is *very* interesting. It could potentially make AJAX programming quite a bit easier and there are some excellent debugging tools for Javascript. One thing I really miss in the Aolserver/Tcl world is a good debugger -- "puts" isn't such a good alternative :D. Javascript, I believe, has nice development environments and debuggers available. It'd also be nice to have a wider world of source code available to us, as Tcl has a lot, but nothing compared to the Perl/PHP world. IMHO, this change could put AOLserver back in the beauty pageant. As always, I welcome any comments or criticisms or flaming of my half-baked off-the-cuff email-to-the-whole-list ideas I presented here. I know there's plenty of them ... :-) I think you nailed it. It's the obvious solution, a very strong direction for aolserver to go. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver's documentation woes and its future
I know that the sad state of the documentation has been a big problem for a long time. I'd really like to hear suggestions from everyone as to how we might go about solving it. I want to chime in with a different viewpoint: 1) the documentation is actually excellent, it's just scattered between old and new, it's just recent features are poorly documented. A few common tasks need how-tos. Most open source projects have little or no docs, only the the massive Apache-sized stuff is a lot better than what aolserver has. 2) there is lots of good competition - everything from Ruby, Python and Zope to LightHttpd is in the same kind of mind space -- alternatives to Apache that have cool ideas in them. 3) the name is a small problem -- I have to explain it to people, but then they get excited by the realization "wow, if aol uses it, it must be pretty good", so while aol is not known for open source projects, it does serve as one hell of a reference customer 4) tcl is more of a problem, people just don't want to learn a language. 5) if there's one thing I think aolserver could do to get more people to use it, it would be to put php in as a standard language, not as cgi, but mod_php style, AND take all the groovy aolserver apis and make them available to php commands. I know I can run php from cgi, but it's in no way better than apache at that task. But... I'll bet that aolserver running php as a primary language would be faster than apache. 6) pick on apache's weak spots, most specifically, languages that are cool but run very slowly in apache. Ruby is what comes to mind -- I'm seeing reports of massive scalability problems with Ruby under apache due to the cgi architecture, and since languages are so easy to glue into aolserver, that seems like a no-brainer. 7) in general, if aolserver was a great place to run the common web languages in a fast, multithreaded way, and did well on the benchmarks vs apache, I think the language zealots would begin pushing aolserver as the best platform 8) BTW, I convinced my ex-company Lyris to give aolserver a try, moving from tclhttpd, and Rusty (who's on this list) is doing a test- port to aolserver right now. So, there definitely are companies willing to experiment with aolserver. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] nsdb dropping handles?
Has anyone ever had any problems with nsdb slowly losing database handles, so that eventually "ns_db gethandle" never returns? I'm working with Vlad to track down that kind of problem, that we're assuming is in his BerkeleyDB/nsdb driver, but I thought I'd ask if anyone's had that problem with other drivers. If I manually "ns_db releasehandle" after every handle usage, I have no problems (but the app is slower), it's only when I rely on the auto-cleanup of handles when the ADP page closes that there's a problem. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl interpreter bloating
Via private email, Rusty pointed me toward the connsperthread, which
does just what I need for my tcl bloating.
It's discussed at
http://www.mail-archive.com/aolserver@listserv.aol.com/msg01525.html
Can I suggest changing the default comment on the default config
file, which is currently this:
#ns_param connsperthread 0 ;# Normally there's one conn
per thread
the "Normally there's one conn per thread" comment seems incorrect to
me and makes this parameter sound like it's about one connection
feeding many web pages at once, which is not what it is at all, but
rather an anti-tcl-bloat feature.
Note also that many modern OSes don't return memory to a process
until a thread exits, in order to minimize semaphore locking around
malloc() (Windows does this, and I think Solaris has added it too)
-john
On Aug 31, 2006, at 10:55 AM, John Buckman wrote:
On my bookmooch.com site, I'm noticing that my nsd process adds
about 300mb of memory usage every day, requiring a restart once a
week after I approach my 2gb memory limit. I'm trying to fix this.
One theory I have is that the Tcl interpreters are slowly bloating.
A simple test.adp page I made with a bunch of tcllib package
requires, followed by package forgets, slowly bloats up.
Apache mod_perl deals with this kind of problem by occasionally
killing an interpreter and reloading it. Is there an automated way
to do the same thing in Aolserver?
I've written this function below, which uses ns_markfordelete and a
namespace global to kill the current tcl interpreter every 100 runs.
namespace eval runtimes {}
proc check_if_should_cleanup_interp {} {
global runtimes::runcount
if {[info exists runcount] != 1} {
set runcount 0
}
incr runcount
puts "Tcl Interp run count: $runcount"
if {$runcount > 100} {
puts "marked tcl interpreter for deletion"
ns_markfordelete
}
}
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
[AOLSERVER] Tcl interpreter bloating
On my bookmooch.com site, I'm noticing that my nsd process adds about
300mb of memory usage every day, requiring a restart once a week
after I approach my 2gb memory limit. I'm trying to fix this.
One theory I have is that the Tcl interpreters are slowly bloating.
A simple test.adp page I made with a bunch of tcllib package
requires, followed by package forgets, slowly bloats up.
Apache mod_perl deals with this kind of problem by occasionally
killing an interpreter and reloading it. Is there an automated way to
do the same thing in Aolserver?
I've written this function below, which uses ns_markfordelete and a
namespace global to kill the current tcl interpreter every 100 runs.
namespace eval runtimes {}
proc check_if_should_cleanup_interp {} {
global runtimes::runcount
if {[info exists runcount] != 1} {
set runcount 0
}
incr runcount
puts "Tcl Interp run count: $runcount"
if {$runcount > 100} {
puts "marked tcl interpreter for deletion"
ns_markfordelete
}
}
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
[AOLSERVER] cron (ie, timed) jobs in aolserver
Anyone have any advice on how to run tasks on a scheduled basis within the aolserver process? Currently, I use cron to run "lynx http://bookmooch.com/ nightlyjobs.adp", and that works fine, but was wondering if there was an all-aolserver way, that was more elegant. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] question about libnsd loading in tclsh
In the 4.5 release docs, it was announced: libnsd: The AOLserver library now includes an entry point suitable for loading into an ordinary, thread-enabled, tclsh, e.g.,: # tclsh % load /usr/local/aolserver/lib/libnsd.so % ns_time Does this loaded libnsd.so share locks, such as ndb_handles and semaphores with an already running nsd? I assume not, but thought I'd ask... What I'm trying to do is run command-line tcl commands that would access my berkeleydb database while aolserver is still running, and if two separate processes are writing the same db file, it gets corrupted. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] ns_cache size-limited by default?
It seems like ns_cache is -size limited by default -- is that correct? This isn't documented, that I can tell. Am I right that the default is: size = 1024 * 1000; It seemed to me, from the docs, that an ns_cache was not size-limited by default, and thus could expand indefinitely. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] ADP flush failed
I'm regularly getting "ADP flush failed" errors on my aolserver console. I assume these appear because the html browser disconnected before the adp page was able to complete. This error doesn't seem like anything so important that it should be displayed on the console screen. Is there a way to disable it? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] cvs head build problem
Cookies require a P3P header now?
Yes, the default security prefs as of IE6 are to discard cookies at
exit that don't have a compact privacy declaration indicating that
you won't share their information with anyone.
Ever wonder why some sites seem to forget your cookies, and some are
incredible at remembering them? The P3P header is the reasons.
I can't prove it yet, but there's some situation that causes ns_db
handles to not get cleaned up. It only happens in production, not
under an "ab" test harness. I solved it by manually releasing all my
db handles and wrapping catch{} statements around everything. It
might be "adp flush failed" "abort exception raised" -- or maybe some
other exception.
Which database are you using? I seem to remember that not all
database drivers
reset after a conn closes.
Vlad's Berkeley DB driver nsdbbd. Stable under stress testing with
"ab" but in the real world, I'm getting handle-not-released problems.
I have my own db api which uses an ns_atclose to cleanup/rollback
anything not
released. In fact, I never release a handle in a script, that is
what the
ns_atclose call is for.
Interesting that you use ns_atclose, why not use the default
mechanism, which is registered with ns_db ? Like so:
static Ns_DbProc dbProcs[] = {
{DbFn_ServerInit, DbServerInit},
{DbFn_Name, DbName},
{DbFn_DbType, DbDbType},
{DbFn_OpenDb, DbOpenDb},
{DbFn_CloseDb,DbCloseDb},
{DbFn_DML,DbDML},
{DbFn_GetRow, DbGetRow},
{DbFn_Flush, DbFlush},
{DbFn_Cancel, DbCancel},
{DbFn_Exec, DbExec},
{DbFn_BindRow,DbBindRow},
{DbFn_ResetHandle,DbResetHandle},
{0, NULL}
};
or is it just the case that you're not using ns_db at all, and wrote
your own?
Catch ends up being a much worse problem than anything else I've
ever used in
tcl. It effectively removes most/all error information, so you
cannot track
down a bug. Also, if a catch is around code which doesn't compile,
the error
message you get is usually completely unrelated to the actual error.
Yeah, I definitely don't like manually releasing all my handles, but
it solves the problem, and when you're in production...
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] cvs head build problem
On Aug 7, 2006, at 6:10 PM, Dossy Shiobara wrote:
On 2006.08.07, John Buckman <[EMAIL PROTECTED]> wrote:
The modifications to util/*.tcl stated above seem critical and the
CVS head should be so patched.
The build is still kinda shaky. In theory, you could build like this
(on Debian):
$ ./configure TCLSH=/usr/bin/tclsh --with-tcl=/usr/lib/tcl8.4
$ make
$ make install
I seem to remember one of the scripts pointing directly to /usr/local/
bin/tclsh as well.
And it'll use the path specified in TCLSH as the executable to use to
execute the .tcl scripts.
I'm not sure I like this: for one, we should probably use autoconf to
auto-detect the tclsh binary using the TCL_EXEC_PREFIX from
tclConfig.sh
or some other magic as a default. That would probably eliminate the
most FAQ build issue with 4.5.0 that you ran up against.
Sure, that'd be better, but as a hack, at least you could make
TCLSH=/usr/bin/tclsh --with-tcl=/usr/lib/tcl8.4
the default, that can be over-ridden. That's not inelegant, I think.
The pre-3.3 gcc error you encountered I created a patch for and folks
have verified it fixes the problem, so I'll commit that now.
Thanks again for the feedback and congratulations on BookMooch.com, by
the way. I love the Server: header. :-)
Thanks!
johnmac:/b/lib johnbuckman$ curl -I http://bookmooch.com
HTTP/1.1 200 OK
MIME-Version: 1.0
P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
Date: Tue, 08 Aug 2006 03:12:07 GMT
Server: MoochServer/4.5.0
Content-Type: text/html; charset=iso-8859-1
Content-Length: 6336
Connection: close
---
btw, that P3P header is pretty much a requirement if you want cookies
to stick around. IE's default w/o that header is to toss them when
the browser exits.
Also, I had a pretty rough day with AOLServer hanging all day after
10 mins of running.
I can't prove it yet, but there's some situation that causes ns_db
handles to not get cleaned up. It only happens in production, not
under an "ab" test harness. I solved it by manually releasing all my
db handles and wrapping catch{} statements around everything. It
might be "adp flush failed" "abort exception raised" -- or maybe some
other exception.
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
[AOLSERVER] cvs head build problem
Tried to build CVS head today on two linux machines, and had to make the following patches to get it to build: inserted #!/usr/local/bin/tclsh to util/nsmakeall.tcl util/nsremove.tcl util/nsinstall.tcl all 3 tcl files were missing a stated interpreter to use and caused the build to fail. I had to "chmod 777" them, too. On the machine with gcc 2.96 I had to # define _nsmayalias otherwise the C code wouldn't build. Gcc 4.1.0 didn't require this. To build the nsdbdb module I had to copy from another build include/Makefile.global include/Makefile.module as these no longer seem to be included in the current AOLserver. The modifications to util/*.tcl stated above seem critical and the CVS head should be so patched. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER]
Now... if I visit a bogus url, I *still* receive the default aolserver not found page. If I visit /404.adp directly, I successfully get the "sorry not found" message. So, the redirects section seems to have no affect at server startup. I've tried a few other methods with the same results. I've gone back and verified that all of this works correctly under 4.0.10. I guess either this is a bug, or just a change in how to handle this. Has anyone else run into this problem? I'd appreciate if anybody could give a bit of time to sorting this out. Yes, I reported this problem a few days ago, and should have mentioned that I was running the CVS head of 4.5, so it looks to me like it may be a genuine 4.5 bug. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Scaling at the high end
that once you cross it, the characteristics of the problem set changes and the optimal (and sometimes only) solutions become very specialized. It's at this end of the spectrum where AOLserver truly shines, but it's a very small set. In general, my experience is that the simplest tool for the job is likely the most scalable, if for no other reason that it's easy to swap out well-understood low-functionality components. AOLServer's easy escalation from Tcl to C for web pages makes for fast initial development, and a straightforward development path to very high performance. My full text index for tile.net (which I sold to internet.com back in 1999) was STL-based C++ wrapped in an AOLServer module, and a memmap-based read-only dbm file sitting mostly in memory. The sort of architecture just isn't possible in most web servers. And, I was able to develop the whole thing in Tcl initially, with less performance critical parts (such as the indexing engine) left in Tcl. AOLServer's internal model is very simple, basically being a C Engine -> C modules -> Embedded Tcl interpreter, with none of the fancy "process watcher" and "shared memory" stuff that Apache has. John, it's fantastic to see you're actively looking at and working with AOLserver. I used to actively use it in the closed-source days of AOLServer. From the list archives, it looks like your first message came through around 15 Apr 2006. I'm hoping that your interest in AOLserver might mean that the upcoming version of Lyris may replace tclhttpd with AOLserver? Sadly, Lyris definitely not being moving to AOLServer, as I sold Lyris a year ago and no longer run the software development dept. The Lyris web interface is all based on tclhttpd. It'd probably just be a week's worth of work to port everything from tclhttpd to aolserver, but they'd need to learn a lot about AOLserver, and list server performance isn't really the hot-button it was in the dot-com era. Way back when Lyris was moving from a mod_perl interface to a tcl- backed one, I wanted to use AOLServer, but the closed-source license was unclear about bundling in a for profit product, and then when things went open source windows server support was not available, so we couldn't use it then. An odd side-note: in my benchmarking of "hello world", tclhttpd gave terrible results vs AOLServer, but a simple all-tcl web server gave amazing results, almost 2x what AOLServer gave: "Hello world" dynamic page on a slow mac-mini: lighttpd-cgi: 15/second tclhttpd: 32/s aolserver: between 640/s and 750/s trivial all-tcl-web-server http://wiki.tcl.tk/15244 : 1162/s Or are you working with AOLserver for something different (i.e., Magnatune)? Magnatune is sadly, all mostly PHP, with back-end Tcl code. I have a new project (www.bookmooch.com) -- a community for used book exchange, which is all AOLServer. And once that launches (in a week or two), I plan on making the german/french versions of the Magnatune web site in AOLServer. One of the reasons I want to use Berkeley DB is that I'd like every web page string to be a BDB database lookup, allowing wiki-style correcting of strings on a web page (ie, anyone can correct, on the spot, any translated text on any page). SQL just wouldn't work with that design, with a dozen or so db lookups per page. -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
