Re: [AOLSERVER] Does not work ns_return + zlib
I'm pretty sure that gzip compression is already natively and transparently
supported in AOLserver as long as you build and include the ns_zlib module.
You shouldn't ever have to bother with encoding or calling a special
function; rather, you just need to enable it in the config file: "ns_param
gzip on" in the ns/server/${server} section of your config file (there's
also a gzipmin param to set the minimum size required in bytes to trigger
compression).
One problem is that AOLserver doesn't gzip content when ns_return is called;
it works for pretty much every scenario except that one. I don't know why
this is the case; we just hacked this by recompiling nsd to always gzip
regardless of the config file, as long as the client supports it (line 161
of connio.c). It would be nice if this worked correctly by default (or if
we figured out the proper way to enable its use in ns_return).
On Wed, Dec 1, 2010 at 9:46 PM, Tom Jackson wrote:
> All you are doing here is using ns_zlib in Tcl. Who cares if this
> works or not? It is possible that HTTP content compression is slightly
> different than what is produced by ns_zlib. But this doesn't matter
> really. What you should be testing is ns_returnz (sp?) or ns_return
> (or adp output) with transparent compression (which takes into account
> client capabilities, you can't just blindly send compressed content).
>
> I remember looking into ns_zlib just because it was part of the adp
> processing/fastpath code. I don't have any reason to believe that, if
> configured correctly it doesn't work. I also doubt that ns_zlib would
> have wound up in the core unless it worked, the author is well known
> and I've never heard of anyone suggesting any of the repo modules
> "don't work".
>
> I'll try to look into how ns_zlib is supposed to work. Maybe this is a
> total lack of documentation problem.
>
> tom jackson
>
> On Wed, Dec 1, 2010 at 10:19 AM, Alexey Pechnikov
> wrote:
> > This work correct:
> > ns_register_proc GET /test ad_test_proc
> > proc ad_test_proc {ignore} {
> > set gzip [ns_zlib gzip "test"]
> > set time [ns_httptime [ns_time]]
> > ns_write "HTTP/1.0 200 OK
> > Content-Type: text/plain; charset=utf-8
> > Content-Encoding: gzip
> > \n"
> > ns_write $gzip
> > }
> > $ curl http://localhost:8200/test 2>/dev/null| hexdump
> > 000 8b1f 0008 0300 492b 2e2d 0001
> > 010 7e0c d87f 0004
> > 018
> >
> > But is possible to disable all output translations by ns_return? Where is
> > documented how ns_return process the data?..
> > --
> > Best regards, Alexey Pechnikov.
> > http://pechnikov.tel/
> >
> > --
> > AOLserver - http://www.aolserver.com/
> >
> > To Remove yourself from this list, simply send an email to
> > with the
> > body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject:
> > field of your email blank.
> >
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <
> lists...@listserv.aol.com> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.
>
--
Hossein Sharifi
http://rateyourmusic.com
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_returnerror 404 and redirects
Hi Brian,
Here's how I understand it to work (anyone with more expertise is welcome to
chime in).
First of all, the redirects in the config file are only used when:
1) nsd handles the relevant case directly (for example, a user requests a
file that doesn't exist and nsd handles it, using your specified 404
redirect).
2) you explicitly call an error proc that is directly related to the error
in question. For example, you handle the "not found" case in your TCL code
and call ns_returnnotfound. You can use ns_returnforbidden to return a 403,
ns_returnunauthorized for a 401, etc. A full list is at
http://panoptic.com/wiki/aolserver/Ns_return
Secondly, the purpose of ns_returnerror is more specific than it might seem
at first. The purpose of that proc is to return a generic "Request Error"
page that wraps the error string that you pass in. It's generally used in
the error cases that do not match any of the predefined errors (like
filenotfound), but you don't have to use it at all if you don't want; you
can just use ns_return with the appropriate status code if you want to
specify exactly what is returned.
So in summary, if you want AOLserver to redirect based on what you've
specified in the config file, you must call the specific ns_return function
that's related to the type of error. Although ns_return and
ns_returnerror can be used to return error-related HTTP status codes to the
browser, they do not utilize the redirects in the config file.
Regards
Hossein
On Mon, Mar 1, 2010 at 10:12 AM, Fenton, Brian wrote:
> Hi Hossein
>
> thanks for replying. Sorry, I should have said that it's not just happening
> with 404s. I tested it with all of the status codes below and none of them
> redirect. So is it the case that you should only use ns_returnerror with
> status codes not in the "redirects" parameter list? I'm a little confused.
>
> ns_section ns/server/${server}/redirects
>ns_param 404"/global/file-not-found.adp"
> ns_param 403"/global/forbidden.html"
>ns_param 500"/global/error.adp"
>ns_param 503"/global/error.adp"
>ns_param 505"/global/error.adp"
>
> thanks for the help
> Brian
>
>
> From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of
> Hossein Sharifi [shar...@rateyourmusic.com]
> Sent: 26 February 2010 17:54
> To: AOLSERVER@LISTSERV.AOL.COM
> Subject: Re: [AOLSERVER] ns_returnerror 404 and redirects
>
> In most cases, you should use ns_returnnotfound to return a 404 error.
> I'm pretty sure that the behavior of ns_returnerror is correct and expected.
>
> On Fri, Feb 26, 2010 at 6:31 PM, Fenton, Brian <mailto:brian.fen...@quest.ie>> wrote:
> Hi
>
> I'm looking at using the 404 redirects in AOLserver. There was a recent bug
> fixed in 4.5.1. where 404s were not redirecting to the config file value
> e.g.
>
> ns_section ns/server/${server}/redirects
> ns_param 404 "/global/file-not-found.adp"
>
> This works great now when I try to access a page that doesn't exist - it
> redirects nicely to "/global/file-not-found.adp". However, when I make a
> call to ns_returnerror with a 404 from within a .tcl page, it doesn't
> redirect to "/global/file-not-found.adp" e.g.
>
> ns_returnerror 404 "This is some error text"
>
> gives me a page saying "Request Error" followed by the text I supplied to
> ns_returnerror.
>
> Is this expected behaviour i.e. a feature or a bug?
>
> thanks
> Brian
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <
> lists...@listserv.aol.com<mailto:lists...@listserv.aol.com>> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.
>
>
>
> --
> Hossein Sharifi
> http://rateyourmusic.com
>
>
>
>
>
> --
> AOLserver - http://www.aolserver.com/
>
>
>
> To Remove yourself from this list, simply send an email to <
> lists...@listserv.aol.com> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <
> lists...@listserv.aol.com> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.
>
--
Hossein Sharifi
http://rateyourmusic.com
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_returnerror 404 and redirects
In most cases, you should use ns_returnnotfound to return a 404 error. I'm
pretty sure that the behavior of ns_returnerror is correct and expected.
On Fri, Feb 26, 2010 at 6:31 PM, Fenton, Brian wrote:
> Hi
>
> I'm looking at using the 404 redirects in AOLserver. There was a recent bug
> fixed in 4.5.1. where 404s were not redirecting to the config file value
> e.g.
>
> ns_section ns/server/${server}/redirects
>ns_param 404 "/global/file-not-found.adp"
>
> This works great now when I try to access a page that doesn't exist - it
> redirects nicely to "/global/file-not-found.adp". However, when I make a
> call to ns_returnerror with a 404 from within a .tcl page, it doesn't
> redirect to "/global/file-not-found.adp" e.g.
>
> ns_returnerror 404 "This is some error text"
>
> gives me a page saying "Request Error" followed by the text I supplied to
> ns_returnerror.
>
> Is this expected behaviour i.e. a feature or a bug?
>
> thanks
> Brian
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <
> lists...@listserv.aol.com> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.
>
--
Hossein Sharifi
http://rateyourmusic.com
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] large file uploads
One possible solution would be to proxy your uploads through nginx (using a recent version). This will not only give you the ability to poll for real-time upload status (and pass that info to the client using AJAX), but it would also allow you to queue the uploads to the backend AOLserver instance one-at-a-time. On Thu, Nov 19, 2009 at 3:20 AM, John Buckman wrote: > On Nov 18, 2009, at 4:34 PM, Hossein Sharifi wrote: > > > Are you using AOLserver 4.5x on a 64-bit platform? nsd will crash on > 32-bit platforms when memory usage (or log files) exceed 2GB. I had to > upgrade for a similar issue involving memory usage. > > > > Although I think that it would be ideal to port the Naviserver code at > some point. > > No, I'm on a 32bit linux. I guess the solution is to upgrade to a 64bit OS > and have way more memory than the max file size. > > I built naviserver today, with their spooler thread, and tested large file > uploads. Naviserver is very efficient at memory when handling the file > upload, but it does hold the entire uploaded file in memory as it hands the > file off to my form handler. So, it has the same peak memory usage as > aolserver, though it needs the peak memory for less time. Still, not an > optimal solution. > > -john > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to < > lists...@listserv.aol.com> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
Are you using AOLserver 4.5x on a 64-bit platform? nsd will crash on 32-bit platforms when memory usage (or log files) exceed 2GB. I had to upgrade for a similar issue involving memory usage. Although I think that it would be ideal to port the Naviserver code at some point. -Hossein On Wed, Nov 18, 2009 at 6:06 AM, John Buckman wrote: > I'm developing a music submission system for Magnatune.com using aolserver > and I'm seeing some problems with large file uploads. > > Specifically, aolserver periodically crashes due to a malloc error. > > It looks like aolserver stores files being uploaded (via form > enctype="multipart/form-data") in memory as they're being uploaded. When > people are uploading zip files of WAV-quality rips of CDs, that's 800mb of > RAM per upload. A few concurrent uploads, and splat. > > It looks like Naviserver has changed the aolserver code in Naviserver > 4.99.1 to > > - Using temporary files for large uploads when content exceeds > pre-configured maxsize parameter > > - New spooler thread for handling large file uploads > > > Does it make sense to port this naviserver code over to aolserver? > > Or, is there another (easier) way to handle large file uploads that doesn't > use lots of memory? > > -john > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to < > lists...@listserv.aol.com> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] AOLserver templating options
Hey all, What's the current state of templating systems in AOLserver? I've found two: 1) ATS: Is it well-maintained, and can it be easily used outside of an OpenACS installation? 2) http://junom.com/document/twt/view/www/ - it seems to be actively maintained given the fact that there are files in that directory created last month, but is it easy to install? And has it been tested in a production environment? 3) Are there any others? Thanks, Hossein -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] AOLServer 4.x blindly reporting X-Forwarded-For value in logs
I've been meaning to write about this for a while: When I switched from AOLserver 3.x to 4.0 a few years ago, I noticed that the IP address in the log file no longer always matches what [ns_conn peeraddr] reports. ns_conn seems to always report the actual IP address of the user, whereas the log files will gladly accept whatever is in the X-Forwarded-For header that the client sends, even if it's forged or nonsensical.This makes it difficult to detect and track bot behavior and other abuses. A significant portion of bot activity on my site is logged as "127.0.0.1," "unknown," "10.0.0.50" or other similar false values. Is there any way to make AOLserver log the real IP address and ignore the X-Forwarded-For header? -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] UTF/asian charset in urls
I don't see this problem in 4.0 or 4.5. It looks like searching for 秋 on bookmooch first goes to http://www.bookmooch.com/search?w=%E7%A7%8B&search.x=14&search.y=13 (which looks fine - that's the correct URL encoding of the UTF-8 representation of that character) but that page immediately redirects to a cleaner search URL ( http://bookmooch.com/m/s/...) which contains the $map(...). Maybe it's related to the code that builds the cleaner URL? -Hossein On 9/28/07, John Buckman <[EMAIL PROTECTED]> wrote: > > Does anyone know what these "$map(%E7%A7%8B)" things are in URLs when > asian-UTF characters are used? > > Ie, typing into a search form on aolserver (BookMooch) for the > japanese character "秋" converts it to > > http://bookmooch.com/m/s/$map(%E7%A7%8B) > > That $map looks like an array lookup, but I've never seen any > documentation about this, or how it might be re-converted back into > UTF. I can't find any mention of $map in the aolserver or tcl source > code, nor online. > > The other possibility is that the hex codes: %E7%A7%8B are just 3- > byte representations of each character, in linear order, that needs > to be reconverted back into UTF somehow. > > -john > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to < > [EMAIL PROTECTED]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] More config questions
These options have moved to the new "ns_pools" interface:
[1153ahost147:~/cvs/aolserver_head/nsd] shmooved% grep maxconns *
...
pools.c:"-maxthreads", "-minthreads", "-maxconns", "-timeout",
NULL
...
Is there any documentation on the ns_pools interface?
The only thing I could find was a section titled "Example: Multiple
connection thread pools" in tests/new/http-test-config.tcl.
But there are several issues with this example:
1) It's commented out. Therefore, I'd guess that it's likely to be out
of date or inaccurate.
2) It only maps two very specific (hypothetical) URLs. How would the
server know how to handle all URLs which did not match these two cases?
There's no documentation for how to specify a wildcard. Is it {*}? or
maybe {}?
3) Most importantly, it doesn't work! I uncommented this example
section, changed the section to match my server name, and placed it in
my config.tcl. The threads for the specified URLs were still limited to
10 threads.
Also, both http-test-config.tcl and base.tcl have
maxconnections/maxthreads in the ns/server/${servername} section.
If there was a working base.tcl available to use as an example, it would
help tons. It would be nice to simply copy base.tcl, change "10" to
"50" and be done with it. I have tried looking at the code in
nsd/pools.c to understand what's going on, but it's a steep learning
curve just to be able to change 4 parameters.
-Hossein
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
[AOLSERVER] AOLServer 4.5: maxconns and ns_returnz
I'm currently having two minor issues with the 4.5.0 release:
Issue 1) No matter what values I set for maxthreads or maxconns, the
maximum number of simultaneous connections seems to be limited to 10. I
don't know if it's being limited by threads or connections.Here's
the relevant part of my config.tcl. I know some of these are
deprecated, but I left them in there just in case.
This is easily reproducable on a standard 4.5 install by running "ab -c
25 -n 1000 http://";. But it also reproduces when I run it as my
real server (therefore, it has nothing to do with the fact that the IP
and URL are identical for each request).
ns_param maxconnections 60
ns_param maxdropped 0
ns_param maxconns 60
ns_param maxsocks 40
ns_param maxthreads 50
ns_param minthreads 40
ns_param threadtimeout 90
Note that I'm not using the new ns_pools feature for now.
Issue 2) Since a lot of the content-type code was cleaned up in 4.5.0,
it's no longer compatible with the nsreturnz module that I (and I assume
others) were using with 4.0.10. However, since 4.5.0 includes nszlib, I
thought it would be easy to write my own version of ns_returnz. It
doesn't seem to work, though.
Note that the 2k requirement is there to temporarily bypass the IE bug:
proc ns_returnz { status type content } {
set enc [ns_set get [ns_conn headers] Accept-Encoding]
set gzip_p [expr -1 != [string first gzip $enc]]
if { $gzip_p && [string length $content] > 2048 } {
set headers [ns_conn outputheaders]
set gz_content [ns_zlib gzip $content]
ns_set put $headers Content-Encoding "gzip"
ns_return $status $type $gz_content
} else {
ns_return $status $type $content
}
}
This code works fine if I return an actual gzipped file as the
"gz_content".
Any ideas on how to resolve either of these issues?
Thanks,
Hossein
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] error: "unable to realloc XXXX bytes"
Dossy Shiobara a écrit : Hmm, interesting. I often upgrade gcc and/or glibc independently of the kernel and haven't run into many issues. You're right - It wasn't difficult to change glibc versions.. and in fact, it was glibc that was at fault. Soon after installing FC5, I installed several update RPMs (including the latest kernel RPM and glibc-2.4.90-19). Downgrading glibc back to 2.4-4 seems to fix the problem - everything's working great now (although I do plan to fix the incorrect usage of exec as well).Thanks very much for the help with this! -Hossein -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] error: "unable to realloc XXXX bytes"
Hi Dossy,
I have tried setting stacksize to 512k / 1MB, and I still receive the
realloc error. I've also disabled vm_overcommit_memory (by setting it
to 2) and it didn't help, unfortunately. For reference, here were the
stats:
$ grep Commit /proc/meminfo
CommitLimit: 2081908 kB
Committed_AS: 326728 kB
$ sysctl vm.overcommit_{memory,ratio}
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
>How much swap do you have configured?
I have 512MB of swap configured, and its usage seems to vary between 0kb
and 150kb. I think 512 is sufficient but I can increase it if necessary.
After some experimentation last night, I was able to get the memory
usage past 1G by running some simple scripts like this in parallel:
set a "1" ; for { set i 0 } { $i < 3 } { incr i } { append a
"11"}
The usage climbed to:
12199 nsadmin 17 0 1591m 1.0g 2984 S 200 34.8 3:37.78 nsd
and it seemed to be stable while running these scripts.
I am beginning to think it's less of a memory issue and more of a
"something's not thread safe" issue (even though i've removed all my
custom modules). I can't reproduce this problem with simulations, but as
soon as I switch my real users to the new server, it begins restarting
within a couple of minutes. The fact that it restarts at around
90-110MB could just be a coincidence.
But I have no idea how to find the culprit. When I switched from
3.3+ad13 to 4.0.10, my server crashed all the time, and I debugged nsd
and found that ns_server was at fault. So I removed references to it,
and everything worked fine.
Now it's crashing in the tcl interpreter, and I have no idea how to get
the underlying tcl code from that (and even if I did, it's probably not
related to the actual cause of the issue, which is more likely heap
corruption from a different thread).
Since all things are otherwise equal between my FC4 and FC5 box, that
might indicate a glibc issue or something similar. But I've found it
difficult to link against a different glibc (since your
bintools/gcc/kernel have to match closely, and upgrading glibc can break
the rest of your system).
Some more info, if it helps: My server receives a medium-high amount of
traffic, about 4 million requests/day, sometimes as high as 200
requests/second.
I don't use ns_server anymore, but I do use [ns_info pageroot], the
"source" command, exec (to run imagemagick, expect scripts, and other
things - probably 1-2 execs/second). I get/set something in ns_cache on
almost every request, although the caches themselves are relatively
small (200-300k), and I believe I was able to reproduce the problem even
with ns_cache disabled. I use postgresql over tcp/ip, 3 pools, 5
connections per pool (I tried Marc's maxidle/open suggestion with no
success). And I don't use ADP at all - only tcl/static content.
Thanks for all your help so far.
-Hossein
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
[AOLSERVER] error: "unable to realloc XXXX bytes"
Hi, I'm trying to set up a new installation of AOLServer (on Fedora Core 5), and I've hit a problem that I can't seem to resolve. The issue is that the nsd process consistently dies (every 20 seconds - 5 minutes) with the error message "unable to realloc X bytes" (some example values for X: 819200, 1005215, 1367511). This might normailly indicate that I've run out of memory - however, the server isn't even close to that state. It has 3GB of RAM, more than 2GB free, and nothing else cpu-intensive running - yet, nsd dies with 95-115 MB of usage (~70M RSS) each time. nsd on my current FC4 typically hovers around 600M (lots of caching) and is stable. I suspect that something in FC5 is limiting the amount of memory a process can use, but I've disabled SELinux and there's no ulimit. I've also suspected a bug in either AOLServer and/or Tcl, but I've tried 3 separate versions of TCL now (8.4.15, 8.4.11, and 8.5.x) each with AOLserver 4.5 and 4.0.10. And of course, I've made sure that threads are enabled in Tcl. I've debugged the process, and the call stack isn't very useful - realloc fails and calls abort() from the call stack of TCL interpretation. Additional info: Stacksize: 2MB Modules: nspostgres-4.0, nscache kernel: 2.6.17-1.2145_FC5smp TCL and aolserver installed with configure/make, (not with RPMs). Top from server, right before crashing: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 3586 nsadmin 23 0 105m 63m 2856 S 15 2.1 0:31.48 nsd Top from my current (non-crashing) server: 20201 nsadmin 17 0 532m 416m 2644 S 2.0 5.2 78:56.84 nsd If anyone has any ideas, I'd greatly appreciate any help. Thanks, Hossein -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
