Re: [AOLSERVER] Segmentation fault
On Thu, 17 Jan 2002 13:15:51 -0600, Rob Mayoff [EMAIL PROTECTED] wrote: +-- On Jan 17, Daniel P. Stasinski said: My second suggestion was meant as humor. The reason that your server crashes is because your code is flawed. Obviously his infinite-recursion example will always fail in some way. But his real code might not recurse infinitely. It might simply need to recurse to a greater depth than the default stack size allows. In that case, increase the stack size is the correct solution. The infinite-recursing code as originally written should work, because catch is supposed to catch errors. And if you run tclsh and try the code, it does the right thing, which is to generate a tcl exception when the recursion goes too deep (which is 1000 levels by default). The problem is really a configuration bug in AOLserver, where the default stacksize is not enough to accomodate the max size of the tcl stack. Two possible fixes: set the default tcl recursion limit lower, or patch the code so that TclpCheckStackSpace() calls NsTclCheckStackSpace(), which is there for exactly this problem. This is done in in the tcl7.6 core, but not 8.x as its just the basic tcl distrib unpatched. -J
Re: [AOLSERVER] metakit with aolserver
On Mon, 1 Apr 2002 22:35:41 +0200, Jean-Claude Wippler [EMAIL PROTECTED] wrote:
The segfault is bad of course, but there may be multiple issues here.
Note that MK defines a mk::set command. So please change from:
namespace eval ::mk {set n [namespace current]}
to:
namespace eval ::mk {::set n [namespace current]}
As I assume that is what you intended?
Yep, that certainly makes the error make sense. Qualifying the set command
fixes that problem.
But AOLServer still doesn't like namespaces. In particular, commands (not
procedures) defined in a namespace. They seem to get lost and I haven't yet
found a way to get to them through 'namespace import' or anything like that.
So I tried an alternate approach, again using renaming: this time, I
changed the mk4tcl code to create commands named __mk__file and so forth,
then from a startup file create tcl procedures ::mk::file et al. Since they
are procedures, they get recreated in the namespace. However, attempting to
use them gets back an error Initialization error in dispatcher. I think
whats going on here is that mk4tcl creates a bunch of c++ objects on
initialization and registers an exit procedure to clean those up when the
tcl interpreter is destroyed; however AOLserver creates an interpreter for
startup and then destroys it and creates more when necessary, so mk4tcl is
left with references to destroyed objects. If this is correct, then either
mk4tcl (or the hypothetical aolserver version, ns_metakit) would need to
call the initialization procedure for each new interpreter (or otherwise
when necessary) or not register the exit procedure.
The namespace thing still bugs me tho.
-J
Re: [AOLSERVER] problem with Linux telnet and nscp
On Wed, 3 Apr 2002 10:12:22 -0800, atfrost [EMAIL PROTECTED] wrote: [-nscp:1-] Warning: nscp: unsupported telnet IAC code 252 received from client This is the WONT command, I believe? That's correct. I haven't dug through the code very deeply, but it bears mentioning that I *can* connect to nscp on AOLServer 3.x, even though that version GetLine() contains similar IAC handling code. On AOLServer 3.x, however, it looks like the session begins with some junk in the input buffer; the first command line always fails. The following message is logged: Error: invalid command name ÿü invalid command name ÿü while executing ÿü The ÿü is the same IAC WONT (dec 255 252) command, just the 3.x nscp doesn't try to grok telnet control commands. Maybe you have something in your .telnetrc that is trying to change settings (linemode perhaps)? Not sure what else would be going on there. -J
[AOLSERVER] Fatal: nssock: select() failed: Bad file descriptor
I'm getting this rather annoying error quite reproducably.
I'm running on linux, aolserver 3.5.2 using nsvhr+nsunix (3_5 cvs tags).
I suspect the error has something to do with http keepalives, even though
nsvhr should be turning those off. The problem never comes up when I'm
browsing my site using IE, but with mozilla as soon as I view a page that
has more than 1 or 2 images on it, the first image gets loaded, then the
nsvhr server spits out this error and hangs (doesn't even die gracefully).
I made the following patch to keep the server running. It tries to check
that a fd is actually open before adding it to the select set, and logs a
bug if its not. Although in mozilla some connections will still hang, the
server now keeps running; but this is a workaround and not an actual fix.
Is there something else that should be fixed? Is there a better way to
check that the fd is valid?
-J
Index: sock.cpp
===
RCS file: /cvsroot/aolserver/aolserver/nsssl/Attic/sock.cpp,v
retrieving revision 1.1.2.1
diff -r1.1.2.1 sock.cpp
719,724c719,729
FD_SET(connPtr-sock, set);
if (max connPtr-sock) {
max = connPtr-sock;
}
if (timeout connPtr-closetimeout) {
timeout = connPtr-closetimeout;
---
if (read(connPtr-sock,NULL,0) == 0) {
FD_SET(connPtr-sock, set);
if (max connPtr-sock) {
max = connPtr-sock;
}
if (timeout connPtr-closetimeout) {
timeout = connPtr-closetimeout;
}
} else {
Ns_Log(Bug,read fails: %s,
ns_sockstrerror(ns_sockerrno));
[AOLSERVER] aolserver bug
I found a bug in aolserver 4.0.10 (and previous 4.x versions, not sure about earlier) that causes the server to lock up. I'm fairly certain I understand the cause, and my fix appears to work although I'm not sure it is the best approach. The bug: when benchmarking the server with a program like ab with concurrency=1 (that is, it issues a single request, waits for it to complete, then immediately issues the next one) the server will lock up, consuming no cpu, but not responding to any requests. My explanation: when the max number of threads is hit then when a new connection is queued (NsQueueConn) it will be unable to find a free connection in the pool and the queueing fails, and the new connection is added to the wait list (waitPtr). If there is a wait list then no drivers are polled for new connections (driver.c:801), rather it waits to be triggered (SockTrigger) to indicate that a thread is available to handle the connection. The triggering is done when the connection is completed, within NsSockClose. NsSockClose in turn is going to be called somewhere within the running of the connection (ConnRun - queue.c:617). However, the available thread is not put back onto the queue free list until after ConnRun has completed (queue.c:638). So if the driver thread runs in the time slice after ConnRun has completed for all active connections but before they are added back to the free list, then it attempts to queue the connection, fails, adds it to the wait list, then waits for the trigger which will never come, and everything stops. The problem is a race condition, and as such is extremely timing sensitive; I cannot reproduce the problem on a generic setup, but when I'm benchmarking my OpenACS setup it hits the bug very quickly and reliably. The explanation suggests, and my testing confirms that it seems to occur much less reliably with concurrency 1 or if there is a small delay between sending the connections. Together these mean that the lockup is most likely to show up in exactly my test case, while much less likely on a production server or with high-concurrency load testing. My solution is to register SockTrigger as a ready proc, which are run immediately after the freed conns are put back on to the free queue (queue.c:645). This fixes the problem by ensuring that the trigger pipe is notified strictly after the free queue is updated and the waiting conn will sucessfully be queued. However I'm not sure this is best: NsSockClose attempts to minimize the number of times SockTrigger is called in the case when multiple connections are being closed at the same time; my fix means it is called exactly once for each connection, or twice counting the call in NsSockClose. It's not clear to me what adverse impact this has, if any, but one thing that could be done is to remove the SockTrigger calls from NsSockClose as redundant. Some additional logic could be added into SockTrigger to not send to the trigger pipe under certain conditions (i.e., if it has been triggered and not acknowledged yet, or if there is not waitin connection), but that would require mutex protection which could ultimately be more expensive than just blindly triggering the pipe. Here's a context diff for my patch: *** driver.c.orig Thu Jan 12 11:39:05 2006 --- driver.cThu Jan 12 11:39:10 2006 *** *** 773,778 --- 773,781 drvPtr = nextDrvPtr; } + /* register a ready proc to trigger the poll */ + Ns_RegisterAtReady(SockTrigger,NULL); + /* * Loop forever until signalled to shutdown and all * connections are complete and gracefully closed. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver bug
You could try my patch; I'm pretty sure it won't have any ill effects (perhaps marginally more cpu use due to a few extra times through the driver loop) but that's hardly any worse than your occasional lockups. Another thing to look at is is you can get a core dump (kill -QUIT) or debugger attached; in my case the driver thread was sitting in poll() and all the connection threads were waiting inside pthread_wait_cond/Ns_CondTimedWait (so of course my first suspicion was a mutex deadlock, but that doesn't seem possible with the structure of the code). -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Application packages
On Mon, 20 Feb 2006 14:52:12 +, Bas Scheffers [EMAIL PROTECTED] wrote: The goal would be to have simple deployable application modules, quite like a WAR file in J2EE. Let's codename it AOLserver Application Package, or AAP for short. (yes, that is Dutch for monkey) This would give the advantage that it would be easy to deploy any open source applications written for AOLserver without having to copy loads of files into the right location or do much editing the global nsd.tcl. Just drop it in and go. You could even do it the Tomcat way; just drop the .aap file in the applications directory and it will unpack and bind it to a url automatically. Could AOLServer use tcl's vfs layer to directly mount say, a zipfile or a tclkit file? This would make for a very nice deployment method and works well elsewhere. Since the configuration file is a tcl program, it could include configuration from other files. Maybe an AAP package could include a file that is to be inserted into the main config.tcl at server startup to load whatever modules and database pools it needs. The big downside is it would need a server restart to load a new package. Whatever you come up with, please don't have the configuration in a few dozen xml files ala tomcat. Just because everyone else does it that way doesn't mean it's a good way to do it. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] perl (nsperl? DBD::AOLserver? etc.)
Dossy Shiobara wrote: On 2006.05.22, 'Jesus' Jeff Rogers [EMAIL PROTECTED] wrote: If there was a DBD interface to the Ns_Db functions that would make aolserver a *schweet* environment for perl apps since there is no database pooling on apache. Oh, interesting. This is definitely going on my TODO list -- DBD::AOLserver ... neat. Of course, testing it will require having the nsperl thing you've done to run Perl scripts inside the nsd. Have you made your stuff publically available somewhere? I haven't previously, but I could. It's not really in the nicest state however (not well documented, sloppy comments (where they exist), etc). I pretty much started from Fontaine's tclperl (v2.3 I think) and added in thread-safety and better handling of SVs (I haven't compared to Jeff Hobbs's work on Tcl.pm) and wrote a little wrapper for use in aolserver. I put it at http://home.diphi.com/users/jeffr/nsperl/nsperl.tar.gz for now, but a more permanent home would be better (once it's cleaned up, naturally). -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver 4.0(.10) memory usage
I think one difference is in the tcl version itself; the latest versions use a fast multithreaded allocator that allocates on average slightly more than double the amount requestes, while earlier versions may have been more frugal. You can cut your memory usage by a third or more by increasing the number of buckets defined in tclThreadAlloc.c and making the spacing closer; there will be a bit more constant overhead and the allocator would run microscopically slower. You may also want to look into the ttrace module which makes all the tcl interpreters lazily load their code instead of loading everything up front which can be a significant savings but I have had very little luck with it (I think the openacs code does too many fancy things to work well with it). -J Guan Yang wrote: Hi, A couple of weeks ago I moved a number of servers to AOLserver 4.0.10 from various versions of 3.4 because of the 10^9 seconds problem. After the move servers that were preivously using 100-120 MB of memory are now using around 300 MB to 400 MB each. I've tried to turn off or reduce adp and fastpath caches, reducing maxthreds to the 40's level. Stacksize is 500k. We have used more or less the same config files, with a few changes like the new nsopenssl configuration format. Has anyone else had experiences with memory usage after switching to AOLserver 4.0, and is there any easy way to reduce it? Thanks, Guan -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver 4.0(.10) memory usage
Guan Yang wrote:
Thanks,
What values would you suggest for NBUCKETS and MAXALLOC?
And looking at tclThreadAlloc.c, I'm not quite sure how to make the
spacing closer.
Guan
The original list is
{ 16, 1024, 512, NULL},
{ 32, 512, 256, NULL},
{ 64, 256, 128, NULL},
{ 128, 128, 64, NULL},
{ 256, 64, 32, NULL},
{ 512, 32, 16, NULL},
{ 1024, 16, 8, NULL},
{ 2048,8, 4, NULL},
{ 4096,4, 2, NULL},
{ 8192,2, 1, NULL},
{16284,1, 1, NULL},
For my experiments I used
{ 16, 1024, 512, NULL},{ 24, 1024, 512, NULL},
{ 32, 512, 256, NULL},{ 40, 512, 256, NULL},
{ 48, 512, 256, NULL},{ 56, 512, 256, NULL},
{ 64, 256, 128, NULL},{ 80, 256, 128, NULL},
{ 96, 256, 128, NULL},{ 112, 256, 128, NULL},
{ 128, 128, 64, NULL},{ 144, 128, 64, NULL},
{ 160, 128, 64, NULL},{ 176, 128, 64, NULL},
{ 192, 128, 64, NULL},{ 208, 128, 64, NULL},
{ 224, 128, 64, NULL},{ 240, 128, 64, NULL},
{ 256, 64, 32, NULL},{ 288, 64, 32, NULL},
{ 320, 64, 32, NULL},{ 352, 64, 32, NULL},
{ 384, 64, 32, NULL},{ 416, 64, 32, NULL},
{ 448, 64, 32, NULL},{ 480, 64, 32, NULL},
{ 512, 32, 16, NULL},{ 544, 32, 16, NULL},
{ 576, 32, 16, NULL},{ 608, 32, 16, NULL},
{ 640, 32, 16, NULL},{ 672, 32, 16, NULL},
{ 704, 32, 16, NULL},{ 736, 32, 16, NULL},
{ 768, 32, 16, NULL},{ 800, 32, 16, NULL},
{ 832, 32, 16, NULL},{ 864, 32, 16, NULL},
{ 896, 32, 16, NULL},{ 928, 32, 16, NULL},
{ 960, 32, 16, NULL},{ 992, 32, 16, NULL},
etc... for I think 108 buckets. My general strategy was to make the
size difference between consecutive buckets no more than about 10% while
keeping the differences as powers of 2. I don't understand the
implications of the maxblocks and nmove numbers as well, so I just
guessed at those.
Another thing you could try is to change your tcl config to not use
tcl's thread allocator at all (by undefing USE_THREAD_ALLOC) and using
google's tcmalloc instead, which seems to take a similar approach with
lots of closely spaced buckets.
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] question about libnsd loading in tclsh
John Buckman wrote: What I'm trying to do is run command-line tcl commands that would access my berkeleydb database while aolserver is still running, and if two separate processes are writing the same db file, it gets corrupted. My understanding (which may be entirely wrong) is that it should be perfectly ok for separate processes to access a bdb file so long as you open it within a bdb environment (which manages locks, transactions, etc). -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver's documentation woes and its future
Dossy Shiobara wrote: It's about why we end up in situations where we even feel a debugger is necessary. I used to use debuggers much more than I do currently, (probably in large part because I used to do more C coding), but even though I do 99% of my debugging without a debugger, that remaining 1% would be far more painful without the debugger. There are 2 main places where I find a debugger helpful. One is when I make stupid mistakes (and if you don't occasionally make stupid mistakes then I doubt you're actually programming) and the debugger can smack me upside the head and point out the stupidity (Oh, it's a fencepost error!). The other, more important case is when I am digging into code that I am not familiar with. In those cases, the debugger can give a new perspective and let you realize what is going on in the code, at which point you can ditch the debugger and go back to the code with a better understanding. Once you are familiar with the code then the debugger becomes much less useful, but in those initial stages it greatly speeds the progress. Linus's rant really rubbed me the wrong way. I thought considered harmful essays went out of vogue a few decades ago. If you'd rather not use a particular tool fine, but don't imply that people who choose to use it are somehow inferior. I thought there was code in AOLserver to support the TclPro debugger - has that been removed? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver's documentation woes and its future
Dossy Shiobara wrote: On 2006.09.06, Rick Gutleber [EMAIL PROTECTED] wrote: As much as it would please, however, me to be able to step through Tcl code, I think Dossy is correct. You really can live without one. I think the point here is, if you force yourself to live without one, it demands that you become a better programmer. Either that, or spend a lot of time chasing down stupid bugs without the debugger. Just because a debugger reduces the time you spend chasing down stupid bugs, it doesn't force you to become a better programmer. When you have a debugger, what's your incentive for becoming a better programmer? Where's the pain that you work hard to avoid? Remembering and carefully avoiding past mistakes (why bother, when you can just root them out in the debugger)? Improving your fundamental design repertoire to eliminate known bad decisions (why bother, the debugger will let you finagle your way through even the worst of rats nests)? Perhaps that's the wisdom that Linus was trying to impart, here. I for one don't think that programming should be painful; I prefer it to be enjoyable (honestly, if I found it painful I'd find a different line of work). Finding bugs with a debugger is plenty painful anyway, and wanting to improve myself as a programmer is enough of a motivation to be a better programmer and not make the same mistakes again. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tcmalloc
Nathan Folkman wrote: What you need to do is build a version of Tcl that has support for threads, but does not use the Zippy allocator. I seem to remember there being issues with core Tcl that did not allow you to do this easily, but you'd have to double check. Maybe try something like --enable-threads --disable-thread-alloc. If I recall from when I was poking at this, there is no configure flag to disable the threaded allocator; instead you need to edit the generated Makefile after you run configure and remove -DUSE_THREAD_ALLOC=1 from AC_FLAGS. And of course, 'make clean' before remaking or nothing will change. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tdom leak fix
tdom can do automatic deleting, but it has some strict limitations. If you pass a variable name to dom parse, as in dom parse $xml doc foreach n [$doc selectNodes ...] ... then the dom object will automatically get cleaned up when the variable doc is unset, as on return from a proc. It does this by setting a variable trace on the passed variable to call delete. The upside is that it gets automatically cleaned up no matter how the proc returns, whether by explicit return or by having an error thrown. The downside is that you cannot return the document thusly parsed from a proc, as the underlying object was cleaned up when the variable it was assigned to was unset. This is an unfortunate result of the tcl object model (that is, Tcl_Objs, not OOP) and is not likely to be fixed before tcl 9 (if at all). -J John Buckman wrote: FYI, on the tdom leak, I found that my app (BookMooch) was leaking as well (thanks for mentioning the leak problem you're having, it helped me find mine), and the reason is that tdom NEVER automatically releases memory for the documents it creates (the docs say the document object command has to be explicitly deleted ). Therefore, I highly recommend adding a $doc delete to manually delete any tdom objects you create. Below are the relevant docs for tdom for this. -john dom parse ?options? ?data? Parses the XML information and builds up the DOM tree in memory providing a Tcl object command to this DOM document object. Example: dom parse $xml doc $doc documentElement root parses the XML in the variable xml, creates the DOM tree in memory, make a reference to the document object, visible in Tcl as a document object command, and assigns this new object name to the variable doc. When doc gets freed, the DOM tree and the associated Tcl command object (document and all node objects) are freed automatically. set document [dom parse $xml] set root [$document documentElement] parses the XML in the variable xml, creates the DOM tree in memory, make a reference to the document object, visible in Tcl as a document object command, and returns this new object name, which is then stored in document. To free the underlying DOM tree and the associative Tcl object commands (document + nodes + fragment nodes) the document object command has to be explicitly deleted by: $document delete or rename $document -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] tdom leak fix
Nick Miller wrote:
Heya,
I have added:
foreach lostdom [info commands domDoc0*] {
$lostdom delete
}
As you say though, best to make sure you catch them when your dealing
with them and explicitly delete them.
Thanks
Nick
Having run into this same problem before I did something similar,
setting up the above as a scheduled proc. Looking at my code I also
included
eval unset [info vars ::dom::_node*Attributes]
although I don't recall if this was for tdom or tclxml.
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] tdom leak fix
Nathan Folkman wrote: Don't you open yourself up for a race condition if that code is in a scheduled proc? If it was an aolserver scheduled proc then yes, but in my application it was running in an after rescheduling loop within tclhttpd so there was no risk of pre-emption; but the memory (dom object) issue was pretty much the same. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_zlib compress
Juan José del Río wrote: aT wrote: I have ns_param gzip on , but do i still need to set some headers ? I don't know. How do i know its gzipping anything ? telnet www.yourwebserver.com 80 And send a GET request. That won't invoke compression unless you include the appropriate Accept-encoding header. You need to sent a complete request like GET / HTTP/1.0 Host: myhost.com Accept-encoding: gzip -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nssession 1.0RC1
Bas Scheffers wrote: Hi Ian, Good to hear someone is using it! I guess I didn't document it very well. You need to pass it a valid HTTP time string. The easy way to get it to never expire is to pass the value of: [ns_httptime 20] That tells it to expire Wed, 18 May 2033 03:33:20 GMT, which ought to be long enough in the future! :) Cheers, Bas. Just in case this is running for 5 or so years, is this susceptable to a Y2038 bug? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_conn outputheaders
Ian Harding wrote: That makes sense, a lot of the headers can't be generated until the body because they are not known, things like Content-Length. I was just hoping to be able to get a nice standard, if incomplete, set of headers that I could send back before streaming. It looks like I can't. Thanks, - Ian There is the mostly undocumented and for backwards compatability only ns_headers command that may do what you're looking for, but you may be better off rolling your own version using ns_write. Ns_ConnSetRequiredHeaders has the comment * Ns_ConnSetRequiredHeaders -- * * Set a sane set of minimal headers for any response: * MIME-Version, Date, Server, Content-Type, Content-Length The set mention there is a good starting place. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Performance problems with large installation
Andrew Piskorski wrote: To reduce your RAM usage, you would definitely want to investigate ttrace, included in the Tcl Threads Extension. However, you don't seem to be running out memory, so this is unlikely to be that big a help in your case. I have had very little success in getting ttrace to work. However, I was looking at the memory stats on mempools.adp and I noticed that the overhead was in the neighborhood of 50%-60%. After a small amount of looking, I realized this is entirely expected: the USE_THREAD_ALLOC version of Tcl_Alloc is a power-of-2 allocator, meaning that it rounds up all requests that it handles to the next higher power of 2. So for example, a request to allocate 600 bytes will really allocate 1024 bytes and the difference is wasted space. On average, you would expect this wasted space to be about 50% more than you requested. So, inspired by the description of google's tcmalloc which uses a large number of buckets, I added a whole lot more buckets spaced closely together (from 8 bytes for the smaller sizes to 512 bytes for larger sizes) in tclThreadAlloc.c and the results were as I hoped; the memory overhead went down to around 10%. Adding all these extra buckets will impose some overhead of its own and a small (probably unnoticable) performance impact, but the amount of memory allocated seems to dwarf this overhead and real impact on memory size is noticable. My server that would be around 90M a few minutes after starting is now only 65-70M. On a memory-tight environment like a unixshell.org setup, this can be quite significant. On a larger server with more memory this can still matter: if trying to keep the overall memory footprint small isn't really your issue then you could think of it as being able to run 40% more threads. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] configured minthreads, maxthreads doesnt show up with [ns_server threads] command
Nathan Folkman wrote: Those parameters moved and are now controlled via the ns_pools Tcl command: This issue keeps coming up. How hard would it be to add in (for the next point release) compatibility code in init.tcl to parse the old-style parameters? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Daniël Mantione wrote: I think a few reasons contribute to the low popularity of AOLserver * It interoperates badly with Apache. Both need port 80. While solutions exits, none is ideal, and none come with Batteries included. Many people (most) cannot rely 100% on AOLserver, despite ocnsidering it superior for web development. I think this may be more of a marketing issue than a technical one. What does apache do that aolserver doesn't? Ok, there are alot of C modules written for apache. How many of these are in high demand? Other than the programming language ones which others are addressing, I'd guess very few, like mod_auth, mod_include, mod_fastcgi, mod_cgi, and *gag* mod_rewrite. AOLserver can do all of these things just fine, although as you say there is no 'batteries included' modules for handling some of them. * It is bad in multi-user environments. You cannot give every use his own space to develop his website in. Actually this problem seems easy to solve, since AOLserver can run multiple instances of itself since 4.0. You can very easily give each user his own space to develop a website in (e.g., ~/public_html) the only problem is if they want to do things as themself rather than as the aolserver uid, since AFAIK setuid and threads do not interact well. A solution could be built using nsproxy with the proxy running setuid as the desired user and sate interps for user ADPs or something along those lines but it would be a fair amount of work that no one seems to be asking for right now. What do you mean by running multiple instances of itself? Back in the old (3.4) days I used nsvhr to proxy to a few completely separate servers running as separate users which worked mostly ok (there were some lingering networking bugs in nsvhr that I was never able to squash) However the server tends to grow in memory size over time and running multiple independent servers just worsens the problem. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Daniël Mantione wrote: I think this may be more of a marketing issue than a technical one. What does apache do that aolserver doesn't? If have had very few situations that could rely 100% on AOLserver. Be it PHP scripts (yes, I know you can install PHP in AOLserver), multi-user requirements or political issues. There is no major technical issue with AOLserver. Not at all. The devil is in the details. There are social issues at work (of which some might be addressable with minor technical interventions). We are in agreement here. This is really the heart of it - it is a social and/or political issue, not a technical one that leads to the poor interoperability. Which is exactly what I said - its a marketing issue. Maybe we (the community) could do the legwork for those technical interventions to address some of those social concerns. Item #1 for such interventions I think would be a apache - aolserver migration guide, not aimed so much at moving a configuration from apache to aolserver (although that would be useful too) as helping an admin who has configured apache previously configure aolserver. the only problem is if they want to do things as themself rather than as the aolserver uid, since AFAIK setuid and threads do not interact well. ... and there is one TCL library, all databases need to be configured globally, cgi scripts cannot be run with user permissions and more. For multi-user systems, Apache is superior. Admittedly its been a long time since I've worked in a highly multi-user environment, but I think these points mostly apply to running external CGI programs, and once you're execing an external progrqam it doesn't matter too much what webserver you're running. For in-process stuff (at least with mod_perl) apache suffers all these same problems, in many cases to a far greater extent. Yes, this is one of the solutions. It can technically be done, in multiple ways, it is even doable, but that is not the point. There is competition on port 80, and you need to have a good story to convince your sysadmin (or find concensus in your open source project) to replace Apache with AOLserver on port 80. Again, a social issue. Again, a marketing problem. It comes down to no one is using it because no one is using it. Idle curiosity - I wonder if anyone is running a system with both apache and aolserver listening on port 80 on different ifs/ips. Should be possible and not even difficult, tho probably of limited utility. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Tom Jackson wrote: None of the issues listed really have a solution. The truth is that if you are doing mass hosting, you should use Apache, the memory footprint is just too great at some point with AOLserver because you have to load each server at startup. At the very least all code for all virtual servers is in memory, at least one copy. Mass hosting of even a hundred domains becomes near impossible. AOLserver cannot be effective in that situation. Apache really is more like sshd, tcpserver, or any other daemon that is just used to startup another process. Mass hosting alot of domains with vastly different setups isn't practical, but hosting lots of domains all configured identically can be done easily, just not with virtual hosts as such. It would need something more along the lines of apache's mod_vhost_alias where parameters for the virtual host are dynamically configured according to rules from a template rather than explicitly. The suggestion from long long ago was to use a custom UrlToFile proc (which sadly can only be done from C code) that is sensitive to the Host header. I'm not trying to be super-advocate boy here, but it just seems like everyone here is making arguments as to why aolserver really isn't good enough compared to apache and it saddens me - if the support community doesn't believe in the product, what chance do I have of convincing my boss next time he wants to shut down that app written in some ancient tcl and aolserver app because its not apache, java, and perl? (to be fair, 3.1 *is* ancient, they're just afraid to let me upgrade it. *sigh*) -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Over in naviserver Vlad (I think) did a bunch of work on vmalloc which tries to actually release the memory back to the system (rather than just the shared pool) on thread exit to keep the process size smaller still. I don't recall what the ultimate outcome of it was however. -J Nathan Folkman wrote: It is possible. ;) There are a number of other things you can do as well to help. One example is configuring your threads such that they die and get reaped after a certain number of requests or time. This will cause memory that is currently tied up in the thread's memory pool to be returned back to the shared memory pool, which can have the net effect of lowering the overall process size. - n On 8/7/07, Daniël Mantione [EMAIL PROTECTED] wrote: Op Tue, 7 Aug 2007, schreef Nathan Folkman: You might also want to try running AOLserver without the Tcl threaded allocator (Zippy). You might want to try Hoard or if on Linux maybe give Google's TCMalloc a shot. Remember, the Zippy allocator is optimized for lock avoidance, and this comes at the cost of greater memory overhead. Yup, I have installations where AOLserver uses very little memory. Try this with Apache :) wwwrun4552 0.0 1.4 7380 3612 ?Sl Jul26 2:21 /opt/aolserver/bin/nsd -ft /opt/aolserver/nsd.tcl -u wwwrun Daniël -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Daniël Mantione wrote: I'm not to happy to call it a marketing issue, because this suggests that if you would have a big enough advertising campain, you can make AOLserver win from Apache. This is not the case. Software lock-in is very difficult to undo (ever try to get someone to change their preferred text editor? mail tool? word processor? webserver ... ) But it *can* be done. Either through convincing them of the benefits (slow and painful), forcing them (not always possible), or providing something that they cannot get elsewhere (e.g., a new framework, access to the company email, or whatever). I don't really mean marketing as in a public relations campaign (although that certainly wouldn't hurt), I mean it more as education and changing the public perception. How to do that? If I knew I'd probably be rich (or a politician). It is the way you need to work with AOLserver that causes these problems. Not because AOLserver cannot do something (on the contrary, it is one of the most capable web servers), but you run into social/political issues, like you needing port 80, have a user who has a MySQL application in his public_html directory, etc. Ok then, first up on the FUQ list (that's frequently unasked questions): Q: How do I run AOLserver when I already have apache running on port 80? A: There's lots of ways, some of which may not work due to your particular political/social situation. AOLserver can do nearly everything apache can, so you could just drop apache. If you have SSI files, [instructions for SSI support] If you have CGI files, [instructions for CGI support with setuid] If you have PHP apps, [instructions for PHP setup] If you have extensive rewrite rules, [script for changing them to filters] If your sysadmin/company/project is unwilling to move away from apache for everything and you still need to coexist, you could: - run both apache and aolserver on alternate ports behind a reverse proxy - run apache up front and use mod_proxy to proxy particular requests to aolserver running on an alternate port - bind aolserver and apache to different interfaces on port 80 - run aolserver on e.g., port 81 and include that in your urls Any others? Should something like this be on the wiki? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Dossy Shiobara wrote: http://panoptic.com/wiki/aolserver/AOLserver_Cookbook#Server-side_Includes I'm only half-joking here--that code I threw up on the wiki was written and tested very, very briefly as a proof-of-concept and only handles the include SSI tag ... it uses a registered ADP tag for its implementation and is likely dangerous to use as-is, but for anyone who wants to bravely add SSI support to AOLserver, it's probably a good starting point. I have some more thorough code for processing SSIs that does the parsing directly, if anyone else needs to do this. I'd need to clean it up a bit before letting anyone else use it; it's in production, just not production quality, if you know what I mean :/ -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Daniël Mantione wrote: Again, a practical situation: How many PHP packages support such headers? (Even OpenACS doesn't support them, so you would have to fix OpenACS too.) Can you (socially) convince those users to rewrite the PHP apps for you to use your OpenACS? As a practical solution, if you needed to do it transparently you'd install a trivial C module to patch the request_rec or Conn and change the peer address to the value given in the proxy header. PHP or OACS wouldn't know the difference. In apache if you have mod_perl installed this is very easy (the C module boilerplace would be 5x the size of the actual code for the module). As for AS, I believe this was done in one or more of the old proxy modules; I don't know offhand exactly what it would be but I'm confident its not at all difficult. What you say about the social problem is very true of course - it takes a whole lot of technical wizardry to overcome the inertia of I don't know that system. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver focus
Dossy Shiobara wrote: On 2007.08.09, Jim Davidson [EMAIL PROTECTED] wrote: After reading through all the responses to my aolserver focus post, it seems to me Thorpe's comments below are the most realistic and actionable, i.e., it's the documentation / getting started stuff that's insufficient. Indeed, Thorpe's message was on-the-spot. As he mentions, there was an effort a while ago where Matthew Burke and I wrote a draft TOC and circulated it to several publishers, but we never actually started real work on the manuscript. I wish I were disciplined enough to complete such a book ... -- Dossy I realize its the lame answer to every similar problem in every open source community, but why not get it started on the wiki? I'd start by cutting and pasting Thorpe's TOC onto a new wiki page, but I'm not clear on what community editing processes (if any) are in place on the wiki. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] binary-clean exec
This is more of a tcl question than aolserver but its coming up in my
server so I'll start here. Is there a way to make exec not do
translations on the returned data? Here's my exact situation. I have a
proc to handle php files by spawning the standalone php program
(this it low-volume so I'm not overly worried about how inefficient this
setup is). This is my proc:
proc runPhp {file} {
set body [ns_conn content]
array set cgienv {
SERVER_SOFTWARE AOLserver/4.0
SERVER_NAME www.example.com
GATEWAY_INTERFACE CGI/0.1
SERVER_PROTOCOL HTTP/1.0
SERVER_PORT 80
REDIRECT_STATUS 200
}
set cgienv(CONTENT_LENGTH) [string length $body]
set cgienv(CONTENT_TYPE) [ns_set get [ns_conn headers] Content-Type]
set cgienv(REQUEST_METHOD) [ns_conn method]
set cgienv(REMOTE_ADDR) [ns_conn peeraddr]
set cgienv(QUERY_STRING) [ns_conn query]
set cgienv(SCRIPT_NAME) [ns_conn url]
set cgienv(SCRIPT_FILENAME) $file
foreach {hl v} [ns_set array [ns_conn headers]] {
set cgienv(HTTP_[string map {- _} [string toupper $hl]]) $v
}
set sli [interp create]
interp eval $sli array set env [list [array get cgienv]]
interp eval $sli [list set body $body]
interp eval $sli [list set file $file]
# binary workaround
interp eval $sli {set pf [open |php $file r+]}
interp eval $sli {fconfigure $pf -translation binary}
interp eval $sli {puts $pf $body}
interp eval $sli {flush $pf}
set result [interp eval $sli {read $pf}]
interp eval $sli {close $pf}
# set result [interp eval $sli exec php $file \$body]
interp delete $sli
set lines [split $result \n]
if {[string match Status:* [lindex $lines 0]]} {
regsub {^Status:\s*} [lindex $lines 0] status
set lines [lrange $lines 1 end]
} else {
set status 200 OK
}
ns_write HTTP/1.0 $status \n
ns_write [join $lines \n]
}
using [open |php r+] works ok, but it makes me nervous about getting
stuck in a deadlock, where the php program is waiting for eof but never
getting it since that won't happen until it closes the filehandle at
which time it is gone. [exec php $body] is much cleaner and works,
except for the things that want to return binary content like images,
and those get mangled by tcl's default line-ending handling.
So - is there any way to make exec use binary translation (i.e., no
translation) in what it returns?
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Working with Chinese characters in Tcl/AOLserver
Janine Sisk wrote: I did find that it does not seem to be possible to open a pipe for both reading and writing. I tried it and was able to write data to the pipe (simulating stdin) but my read from the same channel just hung. Reading from the channel does work if it is opened only for reading. Opening a pipe for both reading and writing does work, but buffering and eof handling can trip you up. Buffering can be turned off with fconfigure or you can explicitly fflush, but eof isn't so simple to deal with - if your inferior process waits for EOF before writing anything then it will get stuck because there is no way to half-close the pipe. If the inferior process writes as soon as it has data available then it will likely work ok. (Incidentally, I asked about this exact same issue about a week ago and got no response; I think it got lost in all the debate about trac). -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Working with Chinese characters in Tcl/AOLserver
Dossy Shiobara wrote: I can't, for example, grab the return value of the command directly from the exec; if I do, it's mangled. I don't think you can tell [exec] what encoding the I/O will be. Perhaps you could/should see if there's a TIP for [exec -encoding $name $command] already ... There is - TIP #259: Making 'exec' optionally binary safe http://www.tcl.tk/cgi-bin/tct/tip/259.html Unfortunately it has been around in draft state for a year and a half now with no apparent action and is targeted for tcl 8.6, so its a ways off. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Working with Chinese characters in Tcl/AOLserver
Janine Sisk wrote: I'm working with a site that stores it's content in big5, and is run through a conversion program to create a gb2312 version for those who prefer the simplified characters. I know these are the charsets being used; I've seen the config files for the converter. Unfortunately the converter was written by a Chinese company with no English info available, does not appear in Google, and is no longer supported even by the original authors. So basically I have to write my own program to do what it does, without any info on how it does what it does. I haven't dealt with chinese characters at all, but this sounds like you're doing character set translations, not character encoding conversions. tcl's 'encoding' command won't help you here - you'd need a monster string map command to change all 6000? code points from one into the other. To draw a much simplified analogy, this is like translating cp1252 to iso8859-1 - you can't do it by simply changing the encoding, you must translate the character set from one to the other by mapping the characters that do not appear in the target character set (in the case of cp1252-iso8859-1 you might map both the left and right single quotes to an apostrophe) The only conversion that works with the java program is to go utf-8 to utf-8s, which it calls simplified utf-8. Google tells me that this is a bastardized format of sorts, proposed by Oracle and not widely accepted. Unfortunately it is, so far, the only one that works. Data comes in as utf-8, gets converted to utf-8s, and goes out through AOLserver configured to use utf-8. All is well. I think simplified utf-8 is the same as regular utf-8 for all code points U+1 (i.e., a single ucs-16 character, which is java's native format for it). So if your encodings are all beneath that you can call it utf-8 without issue. The problem is, Tcl doesn't support utf-8s, and as far as I can tell there is no other format that will work. This will leave me stuck with the java program, and I have serious concerns about the performance of any sort of exec, let alone one that involves writing files. It sounds like the java program is your best bet since it does the translation already; do you have the source to the java program? You might be able to modify it to run better in a pipe, or by being a persistent process so you avoid the fork/exec overhead on every run (e.g., by running it inside tomcat as someone else suggested). If you're really adventurous you could try getting it to run under tcljava but I have no idea if that even works inside aolserver. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Working with Chinese characters in Tcl/AOLserver
Dossy Shiobara wrote: On 2007.09.05, Jeff Rogers [EMAIL PROTECTED] wrote: [...] if your inferior process waits for EOF before writing anything then it will get stuck because there is no way to half-close the pipe. If the inferior process writes as soon as it has data available then it will likely work ok. (Incidentally, I asked about this exact same issue about a week ago and got no response; I think it got lost in all the debate about trac). Perhaps it's time to introduce ns_exec, that yields read and write Tcl_Channel's using pipe()? Good idea? Bad idea? Something you'd like to see in the AOLserver 4.5 tree? +1 (Ok, we aren't apache) I'd rather see something in the tcl core, maybe an implementation in aolserver would work as a proving ground. Are you thinking of lassign [ns_exec somecommand] read_fd write_fd or set result [ns_exec -binary somecommand $input] ? I could see either having advantages in different situations. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Logging
Ian Harding wrote: Is there any sane way to get combined logs from more than one instance of AOLserver? I would like to have the entries interleaved with an identifier of which instance they came from. From what I have read it seems impossible. These instances are on different physical servers too. Thanks, Ian One way to address this issue with apache is mod_log_spread which sends log messages over the spread group communications toolkit[1]. It wouldn't be difficult to set up something similar for aolserver. I don't know offhand if the tcl spread library is threadsafe (I suspect it isn't), but it would probably be better in any case to redo it as a ns_spread module that would share one spread connection across all threads. Still, the api is straightforward and the implementation would fairly simple. Once that's in place just use a filter proc to log to the spread channel. Another route would be to use syslog; I'm not familiar with any tcl libs for that, but again a C implementation wouldn't be too tough. -J [1] http://spread.org/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Logging
Jeff Rogers wrote: One way to address this issue with apache is mod_log_spread which sends log messages over the spread group communications toolkit[1]. It wouldn't be difficult to set up something similar for aolserver. I don't know offhand if the tcl spread library is threadsafe (I suspect it isn't), but it would probably be better in any case to redo it as a ns_spread module that would share one spread connection across all threads. Still, the api is straightforward and the implementation would fairly simple. Once that's in place just use a filter proc to log to the spread channel. I ripped out most of the tcl_spread module and changed it into an AOLserver module that implements a single command with no options, to send a message to a channel. I'd consider it proof-of-concept quality code; there's plenty that could be done to improve it if there's interest. http://home.diphi.com/users/jeffr/software/ -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] New modules on aolserver.am.net; also we are looking to hire a full-time AOLserver programmer
Alex Hisen wrote: We've just finished a complete overhaul of http:// http://aolserver.am.net/ aolserver.am.net/ and our entire company web site. A much better look, better navigation and better organization of content. We've also taken the opportunity to release some new C and Tcl modules: * ADP Master Pages Tcl Module http://aolserver.am.net/code/modules/masterpages.adpx - implements ASP.NET 2.0-style Master Pages under AOLserver 4.0 Thanks for sharing this with the community. It's been somewhat depressing to see every php system include a simple templating system but nothing really solid under AOLserver. (I know OpenACS has a templating system but I didn't look at it long enough to grok all the complexity therin, not to mention that it is tightly tied to OpenACS.) One thought I had while looking at it is that it isn't really suitable for letting untrusted users upload arbitrary master pages for layout/styling/etc, as the users could then run scripts in the pages. I thought to myself, wouldn't a controlled environment for ADPs be nice? So I started looking at what it would take to add in a safe adp execution mode, and I was happy to find that although it's not documented, it already exists! So a suggestion: it would be a nice enhancement to allow for untrusted master pages that are run in safe mode. I'm not entirely sure what behavior makes the most sense (i.e., what can and can not be in safe mode) but the implementation should be pretty simple :) -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks
Tom Jackson wrote: I was looking at lighttpd performance (at http://trac.lighttpd.net/trac/wiki/Docs%3APerformance ) Considering how well AOLserver stands up to lighttpd, my question was why does lighttpd do better? I was wondering if it had something to do with how aolserver does its writes, in particular the Ns_ConnFlushDirect|Ns_ConnWrite - Ns_ConnSend - NsConnSend - DriverSend chain. Each connection thread writes to its own socket and loops to ensure that all the data is written. It seems at first glance that it would make more sense to hand the task of writing to the connection back to the driver thread once the connection thread is done with it, and the single driver thread could multiplex sending over multiple sockets with a single poll loop as well as leaving the connection threads free to do other work. This is the same as the stated rationale for the Ns_QueueWait api - I/O events are cheap so do those upfront instead of having expensive connection threads burdened with wasteful blocking I/O but on the writing side instead of the reading side. Since this is such an obvious change, would I be correct in guessing that it was either tried and discarded as an overall performance loss (there would be alot of data being passed between threads meaning a lot of copying or extra mutexes if the connection thread and the driver thread both accessed the buffer concurrently), or rejected as an incompatible api change (Ns_ConnSend would do something subtly different, and I guess it could make it harder to detect an incompletely sent result)? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] code for 404 pattern (aka static cache)
John Buckman wrote: Bas Scheffers was kind to share his 404-pattern code with me (ie, a custom 404 handler to enable static caching of frequently requested files), which I used to write my own. It's not rocket science, but since I asked the question, I thought I'd share my code, in case anyone else finds it helpful. I've taken the liberty of adding this to the wiki, on the Cookbook page: http://panoptic.com/wiki/aolserver/AOLserver_Cookbook -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] aolserver vs lighthttpd, benchmarks
Andrew Piskorski wrote: What's the actual problem with Gustaf's code? You've obviously read and thought about it, Tom (which I have not), but so far I see a lot of theoretical hand wavy complaints from you, but little solid criticism of the actual code. I also think the code belongs on a module rather than in the core. My reason is because the code is the wrong solution. I'm not saying the code is bad or that it doesn't work, because it obviously works and solves a problem for alot of people. The problem is that with this code, fixing a problem (simple network i/o blocking heavyweight connection threads) requires doing something different at the script level, meaning it is likely that some pages will use the fix and some will not. This is a hack allowing a limitation to be worked around, not a fix for the limitation. When the patch fixes the core so that all connections (or selectively as specified in the config file) receive the benefit, then it belongs in the core. I see this as similar to the inclusion of gzip compression. Several versions ago a module was contributed that added a new command nz_zreturn to allow for gzip-compressed output. This was wrong because someone would need to explicitly choose to use it on every individual page. Including gzip compression in the core is the right way, and that's where it is now. ns_return_background is fine to work around the network writing problem for now, but it should be only a module; the core patch should be a complete fix, one that applies the solution to every connection using plain old ns_return. Yes, we may need to wait a little while for this real fix. And when we do create it, this patch becomes unnecessary except as backward compatibility, which again will belong in a module, for those who need it. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Ns_RegisterAtReady, NsRunAtReadyProcs
Dossy Shiobara wrote: On 2007.10.13, Tom Jackson [EMAIL PROTECTED] wrote: This all looks great, except for the fact that the NsRunAtReadyProcs proc is never called. In 4.5, that appears to be the case. In 4.0, it's called from nsd/queue.c. I'd need to look more closely at the code to see if we should remove those functions entirely in 4.5 or if there's actually a missing call to NsRunAtReadyProcs() somewhere. I'm thinking the call to NsRunAtReadyProcs should be re-added in. It was apparently removed a long time ago, just after 4.0.10 was released: http://aolserver.cvs.sourceforge.net/aolserver/aolserver/nsd/queue.c?r1=1.24r2=1.25 As the api was not being used at all it was fine to remove it, until it became useful for this race condition :) I haven't successfully reproduced the problem in 4.5 yet (or read the code deeply, but the main connection service loop - queue.c:NsConnThread - is not significantly different now) so it may be a different underlying problem than the one I described, but it should be an easy test to see if adding the AtReady callbacks back into the end of the loop helps things. -J But, you're absolutely right: we need to figure this out, one way or another. Thanks for finding this. -- Dossy -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Ns_RegisterAtReady, NsRunAtReadyProcs
Tom Jackson wrote: Now how to get ab stuck and give up? I use a regular browser (mozilla), and I try to load the same image, I hit reload like 5-7 times or so. The number of reloads is larger than the number of maxconns per thread. There is only one thread working. What may be happening is that the keepalive system is expiring the thread all by itself (one client with keepalive). Even though there is a queue, maybe the sock and therefore the conn are being reused, giving no other requests a chance. The last log entry indicates that the thread has processed too many conns and is exiting. Then ab freezes. However, if I reload the page one more time in the browser, ab starts up again. I still think all the evidence points to a logical error of deleting a thread and then not maintaining at least one thread (and prefer minthreads) if there is something in the queue. This isn't a race condition, I don't think, because it is easy to repeat this: it happens exactly the same way every time. So it looks like a keep-wait socket is immediately put to the head of the line looking for more requests from the same client. If you have a web page with 50+ images, you can easily blow out any reasonable maxconns setting, or if someone reloads a less heavy page. Do you still get the lockup in your scenario with keepalives off? If not, it might be a workaround until it gets fixed. If memory serves, it would not be the first time that keepalives have been broken. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Dossy Shiobara wrote: What I'm struggling with though is if I store an image file to the sob with sob.copy and then try and retrieve it later with nsob.get, it only returns the first 4 or 10 bytes of the file. SOB isn't encoding-aware and is actually really naive--it doesn't handle binary data well. Not sure if this it related, but I was just looking at ns_http for an unrelated purpose and it appears that it is not binary clean at all, since it uses Tcl_SetVar which expects a null-terminated string. Getting binary files typically resulted in a 4 byte response, could be that an average jpg file has a null at byte 4. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Michael Andrews wrote: I think the ns_http bug was fixed in the HEAD by Nate. *ashamed* Yeah, the code I'm looking at is 4.0.8, pretty old. I probably just need to upgrade and this will work better. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] openid
Greetings all, Has anyone implemented an openid consumer for aolserver? (or BBAuth, etc.) -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] openid
That'd be great, and I'll be happy to help with the docs where I can. -J Dossy Shiobara wrote: Jeff, I've implemented both an OpenID provider and consumer in Tcl for AOLserver for my blog. I haven't published the code yet because I haven't had a chance to document any of it, but if you'd like to help with docs, I'd be happy to send you a copy. -- Dossy --Original Message-- From: Jeff Rogers Sender: AOLserver Discussion To: AOLSERVER@LISTSERV.AOL.COM ReplyTo: AOLserver Discussion Sent: Feb 6, 2008 6:38 PM Subject: [AOLSERVER] openid Greetings all, Has anyone implemented an openid consumer for aolserver? (or BBAuth, etc.) -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] nssqlite3 troubles
Hi all, I'm using the nssqlite3 driver for my latest experiment and found a serious limitation in it: you cannot execute a select statement that returns no rows. (at least not with select or 0or1row; you could probably hack it with exec but that would be annoying and is supposed to be unnecessary) The problem is straightforward - the driver checks how many rows are returned after executing a statement; 0 returns NS_DML and 0 returns NS_ROWS. 'ns_db select' and '0or1row' expect NS_ROWS results, even if there are no rows to be read (which is a perfectly valid result). So how to fix it? There's no direct way to tell what kind of statement was parsed in the sqlite api (unlike say, OCI) and checking how many rows were changed will give bad results for an update affecting no rows. The dumb thing to do would be to compare the start of the passed sql with select to determine if it is a select or dml statement. But there may be a cleverer way! sqlite allows you to set up an authorization callback that is called when parsing the sql. Among other things this callback can allow or deny operations like select, insert, delete, drop, and so forth. That makes it easy, just create an authorizer to allow select (and read) and deny everything else, and use that for select operations only, and for dml or exec leave the authorizer null to allow everything. The nsdb driver allows for a different selectProc and execProc, so this should be straightforward. Fortunately I read the code before implementing this, for had I not I would surely have slammed my head into the wall several times trying to figure out what was going wrong. nsdb does allow you to register a different selectProc and execProc, and one would think that a select operation would use the more specific function if it was available, and fall back to the generic one if not. One would be mistaken; if an execProc is registered it will be used and the selectProc is only used when there is no execProc. Not only is this backward IMHO, it thwarts a potentially useful protection against sql-injection attacks - if the select method really only let the database execute select statements then adding in a drop table or other nastiness would be ineffectual. However, even if this protection wasn't thwarted at the nsdb level, the database-specific drivers that exist are no help. As far as I can tell, even when a separate selectProc is provided is it just a wrapper around execProc, meaning they all execute whatever sql is passed in and only use the different initial call to check what is returned and raise an error if it is wrong. I gather that these different interfaces were not intended as a security measure, but as a programmer convenience. So after all this I guess I'll shortly be writing a patch for this driver which checks if the statement begins with select and forces it to return NS_ROWS for that case. To whom should I send it? (Or Dossy could grant me cvs access.) And then there's nsdb - I think the more specific selectProc should be tried first for select operations, but since it's been this way for a while, would changing this break some other drivers (where the selectProc has never been called, or tested)? The postgres driver is at least aware of this judging by a comment that the select function is never called by the server, but how would the other drivers fare? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nssqlite3 troubles
Stephen Deasey wrote:
sqlite3_stmt *st;
sqlite3_prepare_v2(handle, sql, length, st, tail)
if (sqlite3_column_count(st) == 0) {
/* DML */
}
Ah, that's much cleaner! I guess late at night I overlooked the
sqlite3_column* set of functions. Using sqlite3_column_name could be a
better approach for DbBindRow also, rather than relying on the per-row
callback being called. Although it may not matter much - does anyone
care about what columns are returned if there are no rows?
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] nssqlite3 troubles
[EMAIL PROTECTED] wrote: The select proc is only in the pg driver in order to support some ancient AOLserver 2 functionality that I doubt anyone else uses any more. Then should it be removed? (in someone's copious free time...) It shouldn't appear in your sqllite3 driver, IMO. Could you expand on this? You've got way more experience with the db drivers than I do so I'm inclined to take your advice, but I'm curious why you think a unified exec is better than separate select/dml/generic functions. Aside from that, do you also think the generic function should be preferred over the specific function if both are defined? I suggest you implement stephen deasy's straightforward check that differentiates between queries that return rows (i.e. SELECT queries but usig SQL Lite's parser) to differentiate between NS_ROWS and NS_DML queries. As far as security goes, no one should allow for the direct execution of external SQL anyway, not even a SELECT. If someone's code breaks because they execute a DROP TABLE statement sent to their site via a query string or whatever, there's not much reason to have sympathy for them. Yes, everyone should check their inputs to avoid this, but things sometimes slip through. What I worry about is a high-profile sql-injection vulnerability being discovered leading to manager-esque types saying AOLserver is inherently insecure because of how it handles database queries, so we can't use it. (It hasn't slowed down php tho.) One of my few gripes about the ns_db interface is that you can only pass raw sql instead of being able to use bind variables. It makes the correct code more verbose and the simple code insecure. I wish I could write my queries as ns_db select $db select * from users where id = ? $user instead of ns_db select $db select * from users where id = [ns_dbquotevalue $user] Yes, it's a straightforward wrapper function to do that (although you wouldn't get the performance benefits from a stored prepared statement that are theoretically possible) but having it as core functionality would be nicer. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Google Summer of Code
Matthew M. Burke wrote: I am convinced we could attract some students, but I don't want to commit unless there's at least a little more positive response. Another possibility is that I know Clif Flynt, Jeff Hobbs and other Tcl folks are putting together an application. So perhaps the better approach would be to list some AOLserver-related projects with their application. What do folks think? An idea I floated a little while back was to make aolserver a competitor in the space of web hosting (particularly, one server hosting many independent domains). The main projects here would be - creating a configuration file wizard (to get up and running quicker) - adding new virtual hosts to a running server - user-setuid operation (that is, let the server read/write files as the owning user of the account, rather than as the server user. This would almost certainly require some kind of proxy process). Executing code as different users would be good too but once you get the file permissions out of the way that's less important. Database access is still an issue tho. The response I got was somewhat chilly in general (if you want apache you know where to get it) tho better from a few people on the naviserver project where they're more interested in trying out radical ideas :) -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] authuser
The authUser field of a conn is logged by nslog and can be read with ns_conn authuser but is only set in the driver from basic authentication. Does anyone (other than me) think there would be value in allowing the conn username to be updated, i.e., by a new 'ns_conn setauthuser' subommand? The primary benefit would be to let you log usernames with requests in your access log when your authentication method is something other than http basic. A possible secondary benefit would be to standardize how different parts of an application (esp. a pluggable authentication system) could pass around the logged-in username. I have a simple patch to add this to stock 4.5; a better overall approach would be to adopt naviserver's more flexible auth handling. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] using openacs db api
Xavier Bourguignon wrote:
Hi All,
Does anybody know what is required to use OpenAcs DB API for
accessing Postgres DB within aolserver?
OpenACS is a complete integrated platform for creating web communities;
I suspect that the DB api builds on the ACS core api and as such it is
not a trivial matter to use the former without the latter. If you're
building a new community system, OpenACS is worth a look, but it's a
large system to just jump into (especially as an intro to aolserver,
tcl, or both).
http://openacs.org/ - OpenACS home
http://openacs.org/doc/current/db-api.html - docs on the ACS db api
That said, it's not difficult to emulate some of the useful things like
bind variables. This proc expands ? in sql statements. It could easily
be made to call ns_dbquotevalue before replacing the variables. It does
not do anything with the db drivers, so it is of no help for something
like bypassing oracle's 4000 character limit as native bind variables do.
proc bindsql {sql args} {
set varcount [llength $args]
set indices [regexp -all -indices -inline {\?} $sql]
if {[llength $indices] != $varcount} {
error [llength $indices] variables expected, $varcount passed
}
set ofs 0
foreach replacement $args idx $indices {
set ix [expr {[lindex $idx 0]+$ofs}]
set sql [string replace $sql $ix $ix $replacement]
incr ofs [string length $replacement]
incr ofs -1
}
return $sql
}
set adminop_sql {select * from tadminop where status = ?}
ns_db select $db [bindsql $adminop_sql 'a']
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Minor facelift to aolserver.com
Juan José del Río (Simple Option) wrote: Yes, but: - But Beta seems to attract so many people to certain technologies/software... Even if some people don't like new people, they are somehow needed. We'll die someday. We need someone to take over our tasks. - We can say that it's 1% beta. Sure there's something that still crashes from time to time! If not, let me upload some patches, and you'll see... lol Note: I am half kidding, half serious. Back in the old days we labeled stuff Under Construction. Beta seems much more formal, snazzy, ... 2.0. http://mpt.net.nz/archive/2005/12/27/web-2 -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Tcl rounding question
William Scott Jordan wrote: Hey all! This is really more of a tcl question, but I'm hoping that someone on the list might have an explanation. Why does [format %.2f 18.005] round down to 18.00 and [format %.2f 1.415] round up to 1.42? Any guesses? Am I missing something obvious here? Tcl version 8.4, if it matters. Thanks! -William Set tcl_precision to 17 to see the fullest expansion of the value that tcl will work with. % set tcl_precision 17 17 % expr 18.005 18.004 % expr 1.415 1.415 % The rounding I think is obvious at that point. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fun, free applications for building personal sites.
Dossy Shiobara wrote: Back in Feburary 2005, Dan Chak at MIT said something to me that I just haven't had the motivation to pursue. However, I really would like to hear what folks think of his statement: What AOLServer community needs is a bunch of fun, free applications for building personal sites. It's stupid, but that's how you get the initial hook on people. I'm not sure who is going to make and release these fun, free apps though, because there's currently also no audience for it. It's kind of a bootstrapping problem. It's definitely a chicken-and-egg problem: most newbies won't try AOLserver for the heck of it but would go through the trouble if there was an application they wanted to run, i.e. OpenACS. Of course, there's little motivation to develop a fun, free app. for AOLserver if nobody's going to try it out. How do we break out of this deadlock? Fun Free apps. The hottest fun free apps currently are mashups and social networking things. And all the data providers (i.e., facebook, flickr, twitter, etc) have php and java client libraries (and occasionally a few others). I don't think any of them have aoslerver/tcl libraries. That is needed as an entrypoint. One starting point I'd be willing to help on here is an openid client (since I've already started work on it). The more traditional personal apps - news/content pages, blog/message board, etc - are still useful too. (and would be a reason for having an openid server also :) Prior to Dossy's call to action on this, I was thinking about starting just such a project for my own use (under the working name ASblog - A Simple blog, or AolServer blog, then expanding it to A Simple Site, but ASS site doesn't seem so catchy). OpenACS is out there; it's a great system but it's a big system, not really the ideal introduction. There's another problem too - free. Or at least, cheap. Not aolserver or the apps themselves, but a place to run them. There are lots of places where you can get php hosting for $5 a month (which includes a bunch of stuff preinstalled, email, etc...). If you want to host aolserver, the cheapest option I'm aware of is an unmanaged linux vps for around $15/mo and *then* you need to set up everything yourself. (and depending on the vps you might have memory issues) I guess this is just part of that chicken and egg problem. Maybe just call it /bin/sh (the shell of the egg... get it?) Somewhat related to this last point is my current interest (and frustration) in helping out small non-profits. I'd like to help out some local groups (preschools, clubs) set up websites for things like fund-raising, school supplies wishlists, news, and so forth; I'd like to use the best technology available as a foundation (imho, aolserver meets that criteria) but the hosting costs make it unattractive - the difference between $60 a year and $180 a year (labor costs are donated of course) is significant at this level. (I vaguely recall a story of Philip Greenspun pitching aolserver/acs to some nonprofit with zero software cost being a big selling point and being rebuffed because in their multi-million dollar budget was already allocated plenty for building completely new systems from scratch. This is a different league I'm talking about. The entire IT budget is maybe a few hundred dollars and most of that probably goes for printer ink.) Which brings me back to something I've mentioned before, shared hosting. AOLserver can do virtual hosts, but a little more framework would be needed to realistically serve multiple independent sites from a single installation. Nothing huge or insurmountable, just some isolation between separate servers so someone doesn't accidentally (or maliciously) overwrite someone else's data. Two quickies come to mind: 'ns_db gethandle' should not be able to get a handle from a different virtual server (I don't know if it can now), and there should be a way to run a safe interpreter and pass commands like 'open' to a setuid ns_proxy. The 3.x era nsvhr/nsunix modules let you run an entire server as a different user, but that would be even more memory-prohibitive. (Before someone says just use apache, it does all this well my point is that I don't want to use apache, I want to use aolserver because it's better for applications, and if it's worse for addressable practicalities when I'd like to address those practicalities and make it better.) -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fun, free applications for building personal sites.
Tom Jackson wrote: Wait, I thought mashups were simply http requests from javascript to update parts of a web page. You return either javascript, html or an xml document, which is probably a worthless option. Maybe I got this wrong, but AOLserver should be able to step into a mashup instantly. What you describe is ajax (asynchronous javascript with xml, although it seems no one uses xml anymore) plus dhtml (dynamic html) and yes, aolserver can certainly provide that without too much trouble. Many (probably most) mashups make use of ajax also, but the mashup part is that they take data from multiple sources - say, from google maps and craigslist - and put it together in some novel new way. A mashup can be done entirely or mostly on the client in javascript but a few things like same-source security, data storage, and having to work entirely in javascript within a browser make it convenient for the server to get involved. Which is why there are the php client libraries I mentioned for flickr, twitter, facebook, etc... The wikipedia entry on mashups gives a better description (and references!) than my blurb here: http://en.wikipedia.org/wiki/Mashup_%28web_application_hybrid%29 -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] bookmooch on reddit
Just noticed that bookmooch is #6 on the reddit homepage. I seem to recall that bookmooch was running aolserver backed by bdb on just one or two servers - is that still the case? What I was wondering more of course is if there was any notable traffic spike from the reddit mention and if aolserver handled any spike without problems. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
While I'd agree this is a bug in fastpath, the real problem is that fastpath is being used at all in this case. The intent of fastpath is to avoid reading a seldom-changed file from disk. It happens to be used in ns_returnfile since that is the normal use case. On unix the fastpath cache is keyed off the dev/inode probably to keep the hash key shorter. Windows doesn't have device and inode numbers so it uses the filename as the hashkey, so it wouldn't run into this problem. From the server side, this could be fixed by: - adding in the filename to the hash key or checking that it is the same - making ns_unlink flush the entry from the fastpath cache - restricting what fastpath will cache - e.g., don't cache anything in /var/tmp or /tmp or a configuration-specified directory. - adding a -nocache flag to ns_returnfile All of these have pros and cons. I don't think your suggestion of waiting for cache entries to age a second or two would work well, it just moves the race condition around and adds a whole lot of disk activity when a busy server is warming up - static files might be read a few dozen times instead of once. Fixing it from the application side is much easier. Just use ns_returnfp instead of ns_returnfile, on the open handle if you generated the file from tcl code and it's convenient to get the handle, otherwise by opening the file right there: ...generate file $myfile in some way... set fp [open $myfile] ns_returnfp 200 text/plain $fp close $fp ns_unlink $myfile You'd probably lose some efficiency by not mmap-ing the file, but that's likely to be noise compared to generating the file in the first place. -J John Caruso wrote: On Monday 01:33 PM 8/18/2008, Tom Jackson wrote: It's not be a data corruption issue because you are choosing to overwrite the old data with new data using the exact same file name. If the data is important, don't overwrite it, thus no corruption. No, you've misunderstood the scenario. The file name needn't be the same to trigger this issue, and the corruption doesn't come from serving data out of a file that's changing, but rather because fastpath caching mistakenly identifies a new file as being identical to a previously-cached file (for the reasons I outlined) and erroneously serves the previously-cached data to the user. This is a design limitation and arguably a bug in the fastpath caching implementation, which is potentially quite serious since it silently serves the wrong data to the user. If you want a more straightforward (albeit contrived) demonstration of the problem, here you go: set file [open /var/tmp/myfile w] puts $file ABC123 close $file ns_returnfile 200 text/plain /var/tmp/myfile ns_unlink -nocomplain /var/tmp/myfile set file [open /var/tmp/myotherfile w] puts $file XYZ987 close $file ns_returnfile 200 text/plain /var/tmp/myotherfile ns_unlink -nocomplain /var/tmp/myotherfile Assuming that /var/tmp/myfile and /var/tmp/myotherfile are created within the same second, the fastpath caching algorithm will misidentify them as the same file, and ns_returnfile will therefore erroneously return the (previously cached) contents of /var/tmp/myfile when it should be returning the (uncached) contents of /var/tmp/myotherfile. - John -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
John Caruso wrote: I'd agree that that's the intent, but the caching is hidden within ns_returnfile and it's not clear at all from the user's perspective that this alligator is lurking in the swamp. Using ns_returnfile in this way may not be the best approach in any particular situation, but it's nonetheless a completely valid usage and isn't contraindicated in any AOLserver docs I've seen. This then is the real fix: mention it in the docs. I put a blurb on the appropriate wiki pages; feel free to suggest something better :) The docs in the distribution should be updated too. It happens to be used in ns_returnfile since that is the normal use case. On unix the fastpath cache is keyed off the dev/inode probably to keep the hash key shorter. Windows doesn't have device and inode numbers so it uses the filename as the hashkey, so it wouldn't run into this problem. No, it can still easily run into this problem--it's just that the file name needs to be the same in both cases (which actually did apply in my client's case, and caused confusion in the early debugging of the problem, since the assumption was that using the same file name and/or path name was the source of the problem). The system needs to be free to do some things to improve performance with the understanding that the user needs to be aware of those things or risk bad behaviour. I wouldn't call it an unreasonable assumption that a file with the same name (and same modtime etc) is the same file. You can run into a very similar problem with NFS (i.e., attribute caching causing a modified file to appear not so) and people have learned to deal with that. - making ns_unlink flush the entry from the fastpath cache Nope, since the file can be removed via (e.g.) exec rm. True, but I'd still put this in the system needs to be able to ... category above. The system does some things and the developer should be aware of those things. I don't think your suggestion of waiting for cache entries to age a second or two would work well, it just moves the race condition around and adds a whole lot of disk activity when a busy server is warming up - static files might be read a few dozen times instead of once. Nope, not at all. The only files that would get read more than once would be those that were served within one second of being generated--which wouldn't apply to any content that fits the definition of static. It would work in your exact case, where the file is always removed immediately after being served and generated. But if not, it would still come up with the wrong answer. 13:50:21 - create file 13:50:21 - serve file (gets cached) 13:50:21 - delete file 13:50:21 - create file again (reuses inode) ... time passes ... 13:55:11 - serve file In this case the file modtime is more than a few seconds old, but the cached mtime, inode, etc. are still matching the file on disk, so the stale cache entry would get delivered. There is also at least one clever optimization where static content does get served within a second of being created, where the 404 page is used to generate something like an image from something like a database and writes it to a file where it is subsequently served by fastpath. So this is actually a fairly non-intrusive fix. The main limitation is that it relies on the file timestamps and the server timestamps being synchronized, which may not always be true. But I can't think of a better solution. Simply put, fastpath caching is inherently broken because it's not possible to guarantee that the file in question really should be served from cache (again, short of a cache-defeating checksum). The same can be said about nearly any caching system: it is unable to handle changes in the data that happen outside of the cache's control or knowledge. This is just the bargain you make when you use a cache. But my point here wasn't to ask about potential workarounds but to highlight the issue itself, since I haven't seen it mentioned before. I think you highlighting it is most of the fix. From there, get the caveat inserted into the documentation and the knowledge into the community so that the next person who runs into this problem will have an easier, or at least less frustrating time solving it. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
Tom Jackson wrote: Think about the case where multiple logical files are actually the same physical file. Using the name would result in caching the same object under different names. This is a much more likely situation than this so called bug. Huh, hard links - I sometimes forget about those. It's a much more believable reason (than my previous suggestion of shortening the key) for why the inode was used instead of the filename for the hash key. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
russell muetzelfeldt wrote: On 19/08/2008, at 9:37 AM, Tom Jackson wrote: On Mon, 2008-08-18 at 15:38 -0700, Jeff Rogers wrote: While I'd agree this is a bug in fastpath, the real problem is that fastpath is being used at all in this case. I don't think it is a bug in fastpath. fastpath is making assumptions about what means something is the same file, and those assumptions are not consistent with unix filesystem semantics - how is this not a bug? It's not a bug because no one ever said that it *was* strictly following unix filesystem semantics, which isn't even a single thing (ufs is slightly different than nfs, is slightly different than ext2 -noatime, is slightly different than afs, etc.) It is following a particular definition: if the file still exists and has the same dev/inode/mtime/size as it did when you last checked, then it is the same file. This of it as a if-modified-since or if-none-match conditional GET. It is a bug in that it's not what you expect. However in that case, the location of the bug is subject to debate. -J sure, the original use case that triggered this seems non-optimal, and could be done in other ways that don't trigger the bug, but that doesn't mean fastpath is behaving correctly... Russell -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
Tom Jackson wrote: No, because each file has a different inode. The bug requires that you create and destroy one file and create another one within one second (so they have the same timestamp) also required that the same inode is used and that the file is the same exact size. But beyond that, hopefully your git checkout will maintain the original timestamp with the file. The bug conditions are actually slightly looser than this, because fastpath checks mtime and not ctime. So a malicious user (or your version control system, if it makes the local files have the same timestamps as those in the repo) could overwrite a file at any point in the future, utime() it back to the same time and fastpath would still consider it the same. So would any number of unix utilities, like rsync, tar, zip, etc. Going back to my previous solutions, the only one on the server side that I still think is reasonable (names break hardlinks, cache flushing on unlink wasn't good in the first place, -nocache - why bother?) is to add a configuration option to exclude particular paths from fastpath. Actually not even a configuration option; that would involve a bit too much overhead for a marginal case; maybe a patch to fix this problem for users for whom it is a problem. Using an unlinked file as a temporary is the right thing to do most of the time, but I imagine ti could be difficult to do when you need to pass the filename around to uncooperative external programs. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
Tom Jackson wrote:
If you want to avoid use of fastpath, just set the configuration lower
than your dynamic content:
#
# Fastpath
#
ns_section ns/server/${server}/fastpath
ns_param cache[set cache 10] ;# max entries ??
ns_param cachemaxsize [set cachemaxsize [expr 5 * 1024 * 1024]]
ns_param cachemaxentry[expr round(floor($cachemaxsize/$cache))]
The description of the parameters here is a little confusing. Browsing
the source, it appears that cache is a flag to enable or disable
fastpath, cachemaxsize is the maximum size of the cache, and
cachemaxentry is the largest size of a file that will get cached.
There is no setting for the max number of entries, the use of $cache in
the settings above (reflecting the server defaults) is really a minimum
number of cache entries (i.e., the default cache will hold at least 10
entries of the max 512k size, but it could also hold 1000 5k files).
I didn't dig deep enough to see how the cache flushing works, but on
casual perusal it looks like the cache is pruned by removing the oldest
entries (not largest, least hit, or least recently hit).
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
John Caruso wrote: Think of it as a if-modified-since or if-none-match conditional GET. Actually that's not analogous, ... I didn't mean to say it was exactly the same, just similar in that given a particular system that makes particular assumptions it is possible to construct a situation where the results are unexpected or incorrect in a particular way. I think by now everyone reading this understands the problem. What's not clear is what you are expecting to happen now. Documentation has been updated to reflect awareness of this problem and caution against using ns_returnfile in this situation and suggesting alternate solutions in the client code. Some code fixes have been proposed, which for various reasons are undesirable or simply won't fix the problem. A default configuration change was suggested which seems generally viewed as undesirable. What more are you looking for? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
John Caruso wrote: BTW, Jeff, the scenario you'd outlined that you thought would trip this up...: 13:50:21 - create file 13:50:21 - serve file (gets cached) 13:50:21 - delete file 13:50:21 - create file again (reuses inode) ... time passes ... 13:55:11 - serve file ...actually wouldn't, because the file would NOT be cached in the second line. The whole point of this strategy is that a file won't be cached if it's been modified within the threshold time (2 seconds in the patch above). Fine, then change that first timestamp to 13:50:18 (say if you ran another external program after creating the file but before serving it that took more than 2 seconds, or if your external program backdated the file mtime.) It's still a race condition that you'll hit if all the stars are in the wrong place. And it still hurts the optimization of using a 404 adp page to generate a heavyweight file only once that gets cached. If your patch solves your problem, that's great, and that's the whole point of OSS. But it does nothing to solve the problem generally and has negative side effects, so I think it would be a mistake to add it to the general distribution. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
Jim Davidson wrote: BTW: Which OS is re-using inodes so quickly? I can't get my Mac OS/X laptop to do that -- figured the inode re-use/prediction thing was plugged years ago, e.g., when fsirand was introduced for scrambling NFS vnodes. Linux. This tcl page: set fn /tmp/tmpfile[expr rand()] set f [open $fn w] puts $f [ns_queryget data] close $f after 2 ns_returnfile 200 text/plain $fn ns_unlink $fn being hit at the same time in 2 windows: $ while true; do res=`curl -s http://localhost:8000/crap.tcl?data=wxyz`; if [ $res != 'wxyz' ]; then echo $res; break ; fi; echo -n . ; done $ while true; do res=`curl -s http://localhost:8000/crap.tcl?data=wxyz`; if [ $res != 'wxyz' ]; then echo $res; break ; fi; echo -n . ; done Will cause one or the other test script to get the error typically withing 5 requests. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
John Caruso wrote: No, you're still not understanding how the patch works. Ok, I'll admit that I misread it at first, but you're also not understanding why I'm saying why it will still break. I'm surprised you're taking such an all-or-nothing view now, I don't think I'm taking an all-or-nothing view at all, I just think your solution isn't the right one. given that you started out being open to discussion. This patch certainly does solve the problem generally--in all but what I'd say are pathological cases, and certainly in any standard usage (like the various examples I've posted). And it does it by directly addressing the fastpath algorithm's reliance on mtime, which has only one-second granularity. pathological cases are exactly the problem. I'm only speaking for myself of course, but I suspect that at least a few others would agree that your case is pathological itself, and not at all standard usage. I can very easily come up with a scenario that breaks your patched fastpath just as easily as the original, to which you can rightly say, but why would you do it that way?. And you would be right. That is the exact same thing that has been said repeatedly on this thread to you: why are you doing it that way? You probably have valid reasons and in any case I'm in no position to question your reasons. That doesn't make your case any less pathological than some other one. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
Derek wrote: how is this thread still alive. I think this bikeshed should be painted blue. See http://www.bikeshed.com/ if you don't understand this. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
Dossy Shiobara wrote: However, to let it continue to spin over and over is unproductive. I'd love to hear other solutions other than configurabe cache key strategies or a time-based delay caching strategy, but the time to debate whether this is a defect or not is officially over: it is a defect. Let's find the right technical solution to make fastpath more robust, please. The solutions I've seen proposed are * configurable cache key + solves the problem for the most common and surprising case - does not solve the OP's problem ? what to make the default? * time-delay + solves OP's problem - still easily broken * flush files from cache with ns_unlink - easily defeated (by using different command to remove files) - links mostly unrelated commands * exclude some paths from caching (e.g., don't cache if the file is outside of pageroot, or is in some configured list of non-cached directories, like /tmp) + keeps temporary files from taking up cache memory - still easily broken - developer needs to know to use excluded paths * add a -nocache flag to ns_returnfile (or add a new command like ns_returntmpfile) to indicate that the file shouldn't be cached + makes intent clearer + keeps temp files out of cache - bloats api - doesn't fix existing cases * change fastpath default to off to be turned on only when desired and understood + simple, effective - default performance hit * use system-provided change notifications to flush cache (e.g., inotify) + effective, efficient - non-portable Did I leave any out? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Data corruption with fastpath caching
Titi Alailima wrote: what you were looking for. Even with the mtime fix there's no guarantee that systems which muck around with mtime (such as tar) won't cause separate files to collide. For a contrived example: I think the best you can do is to use ctime instead of mtime, or maybe btime on *bsd. You can still run into problems if you have clock skew, but there's only so much you can account for. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Help to install AOL server
Thibault Fouache wrote: Hi everyone ! I've got a problem during the Aol server installation at the gmake's step. Here you can find the picture of the error. Can somebody help me please? Thanks in advance, Thibault This is a known issue building aolserver against tcl8.5. Grab the HEAD version of aolserver from cvs; it should build correctly. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to [EMAIL PROTECTED] with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Virtual hosting
You can absolutely run multiple named vhosts off a single IP with no problem. It works out of the box from I think the first release of 4.x. I think the sample config shows all you need to know about vhosting but if not I'd be happy to help come up with a better concise example. the only caveat is that all the vhosts run as the same server meaning the same user. (I had an idea a while back for how to support multi-user virtual hosting via nsproxy, but it never went anywhere) You can absolutely *not* run multiple SSL servers off one ip, and you never will be able to with aolserver or anything else. This is because the certificate is exchanged as part of the ssl handshake which happens before the web server ever has a chance to see it and respond to any Host: header. Re: upgrading - awww, with 4.5.1 just freshly released, doesn't that just make you *want* to upgrade? :) -J Janine Sisk wrote: Hi all, I'm considering moving from owning my own hardware to hosting everything with Amazon. It has many advantages, but one huge drawback - each virtual server can only have one external IP address. I've never tried to use AOLserver's virtual hosting; at one time it was said to be less than reliable, and I've never revisited it. We've always had enough IP addresses that every site could have one of their very own. But that's not going to be the case if I make this change; virtual servers aren't cheap enough that I can set one up for every site, they're still going to have to be roommates. So my question - what is the latest in virtual hosting? Can I actually run multiple sites off of one IP address these days? What about SSL? I'm still using version 4.0.10 - haven't had any need to upgrade. I can upgrade if necessary to deal with this, though I'd rather not introduce that variable at this particular point in time. Thanks in advance, -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOL + SQLite+nsproxy question
Alexey Pechnikov wrote:
I did try to use nssqlite module but get segmentation fault errors. In AOL
server 4.0.10
# tail -f /var/log/aolserver4/offline.sid.mobigroup.ru.log
“db_foreach “select value from session” {
alloc: invalid block: 0xb4cf5378: 78 b4 20
AOL server 4.5 doesn't work with PostgreSQL (nspostgres module produce
segmentation faults of AOL) and I can't use this. I did read code nspostgres
and didn't find any problems... but this code doesn't work. I'm work with
PostgreSQL 8.1 because PostgreSQL 8.3 in debian lenny has datatype errors.
Did you rebuild nspostgres against 4.5 (or better yet, cvs HEAD)? I'm
fairly certain that I've used nspostgres with 4.5.
I also recall fixing at least one segfault in nssqlite3, although that
had to do with releasing a handle that had never been selected on. Or
in other words, make sure you're using the cvs HEAD version on that too.
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_adp_parse issue
Francesco Petrarch wrote:
Ok, I got this, this time I really do, and it should be easily
replicated by anyone.
I have this filter:
ns_register_filter preauth GET /*.html decode_url
ns_register_filter preauth POST /*.html decode_url
ns_register_filter preauth HEAD /*.html decode_url
proc decode_url { why } {
set page [ns_adp_parse -file /www/website/template.adp]
ns_return 200 text/html $page
return filter_return
}
Your problem has nothing to do with ns_httpget.
The problem is that the ADP execution context (which is tied to the tcl
interpreter executing the request) does not get reset until the adp
request completes, which normally happens when the output is flushed at
the end of an adp page. Since you are calling the adp page within a
filter, the full adp page handler is never executed, so the cleanup
never happens, and the adp interpreter is in the aborted state, which
means that it never executes any code.
You can use ns_adp_exception state to get the current exception state
of the adp interpreter. And you can call ns_adp_return to set the
exception state to return which I believe will get turned into ok
the next time through. For example:
proc decode_url { why } {
set page [ns_adp_parse -file /www/website/template.adp]
ns_return 200 text/html $page
ns_adp_exception state
ns_log notice adp exception state is $state
catch {ns_adp_return}
return filter_return
}
It appears that adp code is only expected to be run from a handler, not
from a filter. Perhaps a reset subcommand should be added to
ns_adp_ctl to expose NsAdpReset at the tcl level for cases like this.
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_adp_parse issue
Jeff Rogers wrote:
proc decode_url { why } {
set page [ns_adp_parse -file /www/website/template.adp]
ns_return 200 text/html $page
ns_adp_exception state
ns_log notice adp exception state is $state
catch {ns_adp_return}
return filter_return
}
Whoops, this won't work. The ns_adp_return will let the adp interpreter
run *after* the next eval, but the next time just exits. (an additional
piece of debugging code was hiding this from me initially)
I think a better approach (lacking ns_adp_ctl reset) is
proc decode_url { why } {
set page [ns_adp_parse -file /www/website/template.adp]
ns_return 200 text/html $page
ns_adp_exception state
ns_log notice adp exception state is $state
# reset the adp state
set dn [open /dev/null w]
ns_adp_ctl chan $dn
catch { ns_adp_ctl chan }
close $dn
return filter_return
}
Of course, I had to dig around in the code a bit to figure out how to
call NsAdpReset, but on the plus side the AOLserver code is much more
pleasant to dig around in than many other programs.
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] AOLServer 4.x blindly reporting X-Forwarded-For value in logs
There is no built-in setting to log or not log forwarded headers. It's not difficult to change, but it requires a little programming. If you are comfortable with C, you can edit nslog/nslog.c and change the behaviour with X-Forwarded-For (it's at nslog.c:272 in my copy). You could disable the check and always log the real ip, log both the real and forwarded header, or make it dependent on a config file setting. If you write code that uses a config setting, it might be worth adding to the standard codebase, as I doubt you're the first one to run across this. Alternately, you could do it in tcl code, by setting up a trace filter (with ns_register_filter) that writes out a different log file with exactly what you need. -J Hossein Sharifi wrote: I've been meaning to write about this for a while: When I switched from AOLserver 3.x to 4.0 a few years ago, I noticed that the IP address in the log file no longer always matches what [ns_conn peeraddr] reports. ns_conn seems to always report the actual IP address of the user, whereas the log files will gladly accept whatever is in the X-Forwarded-For header that the client sends, even if it's forged or nonsensical.This makes it difficult to detect and track bot behavior and other abuses. A significant portion of bot activity on my site is logged as 127.0.0.1, unknown, 10.0.0.50 or other similar false values. Is there any way to make AOLserver log the real IP address and ignore the X-Forwarded-For header? -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] is it necessary
nitin chandra wrote: Thank you Everyone, After a lot of attempts i was successful in installing the AoLserver but not PyWX or for that matter any thing which will integrate with Python. Support regarding PyWX on AoL is non-existent, and key people have stopped working / supporting this combo around 8yrs back (at least one person said so). The C-python implementation has a misfeature known as the global interpreter lock (GIL) that presents scalability problems in a threaded environment like AOLserver. I suspect whoever was working on the python integration ran headlong into this, got frustrated, and gave it up. AOLserver is bound to tcl at a pretty deep level, and it does so very well. While there is some support for other languages (php, perl, python, ML maybe?) it will never be as good as the tcl binding. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver templating options
Hossein Sharifi wrote: Hey all, What's the current state of templating systems in AOLserver? I've found two: 1) ATS: Is it well-maintained, and can it be easily used outside of an OpenACS installation? 2) http://junom.com/document/twt/view/www/ - it seems to be actively maintained given the fact that there are files in that directory created last month, but is it easy to install? And has it been tested in a production environment? 3) Are there any others? ADP Master Pages is worth checking out: http://aolserver.am.net/code/modules/masterpages.adpx -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty obvious, although it does result in prequeue filters working differently from all the others, in particular that it would use a different interp from the rest of the request. -J Tom Jackson wrote: There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Jim Davidson wrote: Right -- the pre-queue thing operates within the driver thread only, after all content is read, before it's dispatched to a connection. The idea is that you may want to use the API to fetch using event-style network I/O some other bit of context to attach to the connection using the connection specific storage API. So, it won't fire during the read. Can you share any specific examples of how it has been used? It's always struck me as an unfinished (or very specific-purpose) API since it's undocumented and it seems that doing anything nontrivial is liable to gum up the whole works since the driver is just a single thread. However, a progress callback API could be good -- wouldn't be called unless someone wrote something special and careful enough to not bog things down. Maybe an onstart, onend, and on % and/or on # callback which could do something like dump the progress in some shared (process or machine wide) thinger for other threads to check. I like the idea... Oh, and the pre-queue leak sounds like a bug -- should just be re-using the same Tcl interp for all callbacks. In the case of a tcl filter proc, ProcFilter gets the server/thread specific interpreter for the connection and expects NsFreeConnInterp to be called later, but in the case of pre-queue filters NsFreeConnInterp is never called in the driver thread so it allocates (PopInterp) a new interp every time. Adding in a call to NsFreeConnInterp after the prequeue filters looks like it fixes the problem. If a filter proc is added into SockRead the same thing would need to happen (potentially in the reader thread instead of the driver thread). One thing I am confused about tho, is why without calling NsFreeConnInterp in the driver thread it just leaks the interps rather than crashing when it tries to use the interp in the conn thread, since it looks like a new conn interp wouldn't get allocated in that case. I also don't understand why there can be multiple interps per server+thread combo in the first place (PopInterp/PushInterp); I'd expect that only one conn can be in a thread at a time and that it always releases the interp when it leaves the thread. -J -Jim On Nov 25, 2009, at 5:46 PM, Jeff Rogers wrote: It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty obvious, although it does result in prequeue filters working differently from all the others, in particular that it would use a different interp from the rest of the request. -J Tom Jackson wrote: There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
The YUI upload control looks like a good place to start for the flash client-upload feature. I haven't looked into it too deeply tho, so I don't know what the server side looks like. YUI Uploader widget: http://developer.yahoo.com/yui/uploader/ Other that that, I was pondering the plain upload issue. Since IE/Chrome/Safari are timing out on the upload, I wonder if the connection could be kept alive by sending something - anything - back to the client while it is still uploading. This might be doable with Jim's new read filter. Of course, the browsers might respond to data by closing their connection or stopping sending, or crashing (you never know with IE). And then even if it works, you have the problem of not having the tcp connection interrupted for however long it takes, which can be iffy in the world of flaky wireless connections and ISPs. -J Tom Jackson wrote: This method could also have the advantage of recovery in case of a failed upload. A client would look much like a udp application which tracks packets at the application level. Once the server side API is set, the client could be javascript, java, flash or tcl. The client-side solution also has the advantage of being tailored to the website, and you could use more bandwidth-efficient compression and/or binary transfer instead of encoded transfer. But if you want to provide upload progress to your customers, it seems counterproductive to create a client which queries the progress via a separate url. That just greatly multiplies the number of requests the server must handle. tom jackson On Mon, Jan 18, 2010 at 2:39 AM, John Buckman j...@bookmooch.com wrote: On 11/24/09 5:13 PM, John Buckman wrote: Is there any access (in C or Tcl) to an upload-in-progress in aolserver? It'd be nice if we extended ns_info with [ns_info driver ...] that could give you connection-level info. from the driver thread. In its simplest form, all we need is to expose the total bytes read/written on a socket from the driver thread. Bytes read of the POST request's body and the anticipated Content-Length enables us to compute a rough progress - using the unique URL bit gives us an opaque handle to identify which connection we're interested in. I've learned a few things by deploying a large-file-upload feature on aolserver: 1) IE times out on large file uploads over DSL, as does Chrome and Safari. Only Firefox seems to have a long enough timeout to enable 600mb file uploads over DSL. 2) All the other file upload sites use a client-side widget to upload a file in parts, not using the browser's upload feature at all. Then, they have a thin server-side program which accepts small chunks of the file upload at a time. Once the widget decides the entire file has been sent, it submits to a new web page, which then collects all the small file chunks. So... instead of working on an upload-in-progress feature, it would make sense instead to have a client-side widget (javascript/flash/java) that sends file upload chunks to a server-side tcl script, and then have a harvester tcl script once the widget says the file upload is complete. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver on GitHub
Dossy Shiobara wrote: Hi, So, I've finally gone crazy and have started pushing the contents of the CVS repository from SourceForge up to GitHub: http://aolserver.github.com/ http://github.com/aolserver I think in general this is a good thing. In fact, I was just thinking about doing the same thing myself, except self-hosting with fossil instead of git. :) (no holy wars please) While you're making updates, some of the links on www.aolserver.com are bad - the 'Tcl API' link ends up point to dev.aolserver.com which is apparently dead; some other stuff is flaky or slow (the panoptic.com wiki with the good docs often seems sluggish). If you could fix up a few of these broken-ish links, that'd be nice. Cheers, -J I'm hoping this will make it easier for people to fork the code, make their own individual changes, and share those changes with the rest of the community. I'm hoping this will effectively eliminate any previous barriers to contibution - fork the code on github as much as you'd like in your own account, share your changes with whoever you please. Is this an official change? Of course not! The code still lives in CVS at SourceForge, like it always has - but now a copy with full CVS history is up at GitHub, as well. At this point, what does it mean to be official anyway, right? As of right now, I've only imported the aolserver module from CVS into GitHub. Over the next several weeks, I plan to import all the modules from CVS, but this takes time. A nice feature of GitHub is the included Issues system, their Wiki, and their static page hosting. If I get really ambitious, I might look to move as much of the AOLserver documentation as I can into the GitHub Pages area. As usual, feel free to flame me for running off into the weeds and just doing something without getting consensus or involving the community but I'm hoping at least a few of you will find this work worthwhile. If you have any positive comments and/or suggestions, don't hesitate to get in touch with me: I'd love to hear what you think. AOLserver's not dead, yet. ;-) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver on GitHub
Dossy Shiobara wrote: Hi, So, I've finally gone crazy If you have any positive comments and/or suggestions, don't hesitate to get in touch with me: I'd love to hear what you think. AOLserver's not dead, yet. ;-) Also while you're at it, is the chat logger still running? And would anyone else be interested in moving the official aolserver chat from irc to a more modern and functional jabber server (perhaps with a bridge, like the tcler's chat has)? -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] WebSockets in AOLserver: get the full request body
w...@dcon.at wrote: Hi! So it's simply not possible to retrieve the 8 bytes after the header? That's a pitty, because that would be all I need for implementing the WebSocket protocol and with ns_chan one could feed data to a lot of clients. wiwo I'm pretty sure it's not possible to implement websockets in AOLserver without some C-level work, and if you could you wouldn't want to. You can't, because the driver reads the entire request, including content, before handing it off to the connection thread. And it only reads past the headers if there is a 'Content-length' header. (You can hook into the content-reading with a read filter, but that won't help because with no content-length it won't ever try to read it). You wouldn't want to, because you would attach a conn thread to a single persistent websocket, which would severely limit how many you could handle. Doing it as a background tcl thread would be little better, because you'd still need a thread per connection. Handling all websockets with a single background thread using tcl event handling would work, if you could get read/write handles to the sockets (which currently you can't - I think the background delivery patch only creates a writable channel). So what else can you do? For starters, you could check the current state of naviserver - I'm not up to date on what has changed there lately, but a r/w version of 'ns_conn channel' might have been implemented. Otherwise, you need to get your hands dirty in the C code and implement 'Upgrade' handling. If you do take this approach, I'd suggest implementing just enough logic to transfer control elsewhere - that is, in the driver, don't try to handle websockets, rather set it up to recognize the upgrade protocol (Connection: upgrade and Upgrade: something headers) and call a registered procedure and forget about the connection (leaving it to the registered proc to actually do anything with it). -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] WebSockets in AOLserver: get the full request body
It looks like my previous answer was, at best, partially right. It looks lke [ns_conn channel] does in fact give you a read/write channel. The effect of the missing content-length header is to discard any data that was read past the headers. Attached is a patch (readahead.patch) to change that behavior. The patch does 2 things: one, if the request includes 'upgrade' in the connection header, it doesn't read past the header (and doesn't throw away what it has read). Two, when [ns_conn channel] is called, it will insert into the channel buffer any data that was already read (and not processed) on the socket. Combined, it means that on an upgrade connection you can get the entire body, even if there was no content-length header sent. Also attached is a tcl file (index.tcl) that looks for an upgrade header, and when it gets one it forks a background tcl thread that echoes everything read back to the sender. Every new request will get a new thread. It also probably needs some tweaking to get a second request to work at all (since it's using shared channels to give the socket to the background thread). Importantly, this design of thread-per-connection is not scalable - your server will fall down and start crying once you start up a few dozen threads. (perhaps more; a really rough guess is that each tcl thread consumes 2M of memory before doing anything). What you really want to do is to transfer each new websocket to a single existing background thread, but I'm not sure how to send stuff into another thread's event loop using the aolserver thread tools. The standard tcl thread package is more powerful (with thread::send and thread::wait) but I don't know if those play nice inside AOLserver. -J wiwo wrote: Hi! I just tried the following: I've been telneting to our server and was feeding it this request: GET /test/ns_chan HTTP/1.1 Host: www.dcon.at Connection: Upgrade Sec-WebSocket-Key2: 12998 5 Y3 1 .P00 Sec-WebSocket-Protocol: sample Content-Length: 8 Upgrade: WebSocket Sec-WebSocket-Key1: 4 @1 46546xW%0l 1 5 Origin: http://example.com ^n:ds[4U Notice the Content-Length header. As Jeff told me, I've now been able to output the 8 byte body of the request. Then I was creating an ns_chan with: set ch [ns_conn channel] ns_chan create $ch $chan_name and getting it back with: set ch [ns_chan get $chan_name] The I tried to write my WebSocket server handshake with: puts $ch HTTP/1.1 101 WebSocket Protocol Handshake Upgrade: WebSocket Connection: Upgrade Sec-WebSocket-Origin: http://[util::host] Sec-WebSocket-Location: ws://[util::host]/[ns_conn url] Sec-WebSocket-Protocol: sample $ret which showed up in the telnet client. Then I tried to read again set str [read $x 10] ns_log notice $str and typed 10 characters in the telnet client which showed up in the log. So far so good! I could now write my own proposal for changing the WebSocket Client and make Google and Apple change their implementations of the protocol. Unfortunately my relationship with Steve and Larry cooled down the last few months, so this could take a while :-) Or I could get my hands dirty and mess around with the code in driver.c. As far as I can see, the crucial code is in static ReadErr SockReadLine(Driver *drvPtr, Ns_Sock *sock, Conn *connPtr) somewhere at line 1878 where the content length is read. I could simply check for the Upgrade: WebSocket header and set the length to the magic number of 8. Of course I have no idea when I'm talking about AOLserver C code, but maybe someone with a clue can give me his/her thoughts here. wiwo I'm pretty sure it's not possible to implement websockets in AOLserver without some C-level work, and if you could you wouldn't want to. You can't, because the driver reads the entire request, including content, before handing it off to the connection thread. And it only reads past the headers if there is a 'Content-length' header. (You can hook into the content-reading with a read filter, but that won't help because with no content-length it won't ever try to read it). You wouldn't want to, because you would attach a conn thread to a single persistent websocket, which would severely limit how many you could handle. Doing it as a background tcl thread would be little better, because you'd still need a thread per connection. Handling all websockets with a single background thread using tcl event handling would work, if you could get read/write handles to the sockets (which currently you can't - I think the background delivery patch only creates a writable channel). So what else can you do? For starters, you could check the current state of naviserver - I'm not up to date on what has changed there lately, but a r/w version of 'ns_conn channel' might have been implemented. Otherwise, you need to get your hands dirty in the C code and implement 'Upgrade' handling. If you do take this approach, I'd suggest implementing just enough logic to
[AOLSERVER] How about fossil? (Was: Git on SourceForge)
Jade Rubick wrote: Unless we hear otherwise, so far I think we can summarize this thread as: Tom strongly dislikes github. Several other people favor it. The rest don't care or haven't spoken up yet. I'll toss in my 2 cents. For my recent projects I've begun to use fossil. It has a distributed wiki and bug tracker in addition to distributed source control. The command set is very simple, much simpler than git. There is no equivalent of github for fossil, but it doesn't need one; fossil includes its own web interface that runs as a cgi. The big win of fossil is that installation is simple; it's one executable that includes everything (command line tools, client, server, ...), and it works the same on unix and win32. It's not for everyone, but I highly suggest taking a look at it. And I wouldn't mind if aolserver started using it :) (I've been meaning to set up my own aolserver fossil repo, but I keep not finding the time to do so.) http://fossil-scm.org/ -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Does not work ns_return + zlib
Hossein Sharifi wrote: One problem is that AOLserver doesn't gzip content when ns_return is called; it works for pretty much every scenario except that one. I don't know why this is the case; we just hacked this by recompiling nsd to always gzip regardless of the config file, as long as the client supports it (line 161 of connio.c). It would be nice if this worked correctly by default (or if we figured out the proper way to enable its use in ns_return). I only skimmed over the code, but it looks like the problem is that the content is only gzipped if the NS_CONN_GZIP flag is set on the conn, and the only place that flag is set is in the adp processor. The obvious (I think) improvement is to expose Ns_ConnGetGzipFlag/Ns_ConnSetGzipFlag as subcommands of ns_conn, which would let you write a filter to do the ancient/broken browser checking or whatever else you needed and turn gzip on or off as needed. (However, doing this could impact adp processing, so a corresponding change might need to be made there, lest your adp code has a different idea about what should or shouldn't be compressed) -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver, nginx, and port numbers
It doesn't seem to be documented, but you can set a 'location' parameter for nssock that will override this: ns_section ns/server/server1/module/nssock ns_param port 8082 ns_param location http://shared.furfly.net Cheers, -J Janine Ohmer wrote: My problem is the value returned by [ns_conn location], which is http://shared.furfly.net:8082. This is partially correct; I told the site it's hostname is shared.furfly.net and it seems to be ok with that. But I can't get away with telling it that it's running on port 80, and [ns_conn location] seems to be returning whatever's in the config file. What I want, of course, is just http://shared.furfly.net. Is there some way to set this up so it returns the desired value, even though from AOLserver's point of view that's not exactly correct? -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] What does 'exiting: exceeded max connections per thread' mean?
Haig Didizian wrote: Hi there, Is there any wisdom in this community about whether it's better to let these threads run forever, or whether it makes more sense to kill them off after they process a number of connections? As others mentioned, the downside of letting a thread run forever is that a memory leak could cause the server to grow, so killing off the threads after some number of requests is a reasonable safety precaution. The downside of killing off the threads is that it takes a nontrivial amount of time to start up a new thread; letting the thread server multiple requests amortizes this startup time across all the requests. So it's a trade-off between performance and safety. Naturally, the best way to find good values is to do the benchmarking yourself. From what I recall the last time I did my own benchmarking there was very little additional performance to be had once you're serving 1000 requests per thread, although this could be higher if you have lots of library code. Interesting side note: while lowering the number of conns per thread would seem to increase safety/stability at the cost of performance, I've found that lowering it too much (i.e., to 1 or 2 requests per thread) makes the server very unstable, often segfaulting after a handful of requests in quick succession. Raising it to 15 or more requests per thread and I haven't seen any problems no matter how hard I hit the server. This surely points to some kind of race condition, but I haven't figured out what it is. I suspect it's just the server slapping an admin intentionally trying to cripple it :/ -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Feature request: ns_querygetarray function
Alexey Pechnikov wrote:
2011/5/10 Dave Bauerd...@thedesignexperience.org:
I am not sure I understand hat you are trying to accomplish. What is an HTTP
Form Array? I am not aware of such a thing. It looks like you are passing
some application specific formatted data and need to process it.
This is very popular situation! The HTML example with checkboxes array:
It may be a common situation, but that doesn't mean it is well-defined.
In your example below, what would querygetarray do when the counts
of the keys are not the same? Would it generate blank entries for
those, drop the keys without enough values, or refuse to process it
(possibly by raising an error)?
A fuller specification of what the function does would be helpful. Does
this function exist for other frameworks? Maybe seeing the docs of
those would help explain things.
Otherwise, this seems like a function specific to using jquery; perhaps
it would go better in a jquery support library.
-J
form method=post action='';
input type=checkbox name=box value='John'
input type=checkbox name=box value='Mike'
input type=checkbox name=box value='Rone'
Param box is array.
As example, a simple HTTP GET request:
/?a=firsta=secondb=1b=2
In this request we can see 2 two-elements arrays:
a=first, a=second
b=1, b=2
or 2 rows:
a=first, b=1
a=second, b=2
Problems:
1. When names and number of the array elements are not known we can't
use ns_querygetall
2. When some items can be not defined is needed the additional check
of arrays elements count. As example
/?a=firsta=secondb=1a=thirdb=3
But
[ns_querygetall a] {first second third}
[ns_querygetall b] {1 3}
Of course, these are _not_ the 3 rows becouse value [lindex
[ns_querygetall b] 2] = is invalid!
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Problem uploading HUGE files. Possible AOLServer bug
You could set the max upload value to INT_MAX (2^31-1, as you have
below), and that should work, but going higher will be difficult.
Simply changing the variable to a long in one place is insufficient,
because that value is handled in quite a few places, and you'd need to
change all of them; the obvious one is the Limits structure referenced
in the code snippet you have, but there's also buffer sizes and scanf
formats (to read the upload size from the http request) to handle, and
probably a few others as well.
-J
Eduardo Santos wrote:
Hi everybody,
My system have very specific needs to upload really large files
sometimes. My upload limit is actually 400MB, but I've received a
request to raise this value to 2GB.
I'm using AOLServer 4.5.1 and I know this is based on ns_limits command.
So, I used the following piece of code to raise the upload size:
set max_file_upload_mb 2500
set size [expr $max_file_upload_mb * 1024 * 1024]
ns_limits set default -maxupload $size
However, the response was completely unexpected:
nrunning 3 nwaiting 0 ntimeout 0 ndropped 0 noverflow 0 maxwait 100
maxupload -1673527296 timeout 60 maxrun 100
/maxupload/ had a negative size! I started digging up in C source code
from AOLServer, and I guess I've found the problem. In the file
nsd/limits.c we have the following piece of code:
int i, val;
static CONST char *opts[] = {
get, set, list, register, NULL
};
enum {
LGetIdx, LSetIdx, LListIdx, LRegisterIdx
} opt;
static CONST char *cfgs[] = {
-maxrun, -maxwait, -maxupload, -timeout, NULL
};
(...)
switch (cfg) {
case LCUploadIdx:
limitsPtr-maxupload = val;
break;
}
}
It seems like /maxupload/ receives the value from /val/, wich is an int
var. If I'm not wrong, the maximum value for C int vars
is 2,147,483,647, and I was trying to setup this value to 2,621,440,000,
wich seems to be causing an overflow.
Considering all this, if I just change the line below everything should
work, right?
int i, val;
I could just use:
int i;
bigint val;
Did I say something wrong or this line of thinking make any sense?
--
Eduardo Santos
Analista de Sistemas
http://eduardosan.wordpress.com
http://twitter.com/eduardosan
-- AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the
email message. You can leave the Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] 2Gb log file problem
The log module supports rolling. Normally it is done daily, but if your logs are growing too quickly and hitting 2G within a day you could set up a cronjob to send the server a signal to roll the logs, or a scheduled task within the server to roll the logs every hour, or when they reach a certain size. The command to roll the logs is [ns_accesslog roll] -J Alexey Pechnikov wrote: With 2Gb size of log file all scripts don't work and the log file don't growing. Only static files are serves -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 2Gb log file problem
Alexey Pechnikov wrote: 2011/5/17 Jeff Rogersdv...@diphi.com: Undocumented behaviour is bug, isn't it? I didn't suggest that it wasn't, I only offered a possible workaround for you to solve the immediate problem facing you. There is a bugtracker for aolserver at sf where you can file bugs, although there's no guarantee ho quickly any of them will get looked at or resolved. http://sourceforge.net/tracker/?func=browsegroup_id=3152atid=103152 -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 2Gb log file problem
Alexey Pechnikov wrote: Thx, I use daemontools for starting AOL Server /usr/sbin/aolserver4-nsd -f ... and stdout may be logged by multilog (as example). Oh, the server log has the same issue. That one doesn't have automatic timed rolling, but it will still roll on a signal or an appropriate command, in this case ns_logroll which could be set up as a scheduled proc. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Problem uploading HUGE files. Possible AOLServer bug
Eduardo Santos wrote: Hi Jeff, Thank you for the answer. I was thiking that such a problem really exists, and that's why I decided not to touch the source code on this. It seems to me a bad thing to have a such low limit for file uploads. As I don't know this implementation, do you think there's any other way we can increase the file upload limit? I think someone else was looking at uploading large files last year. The problem they were running into was that unless your link is fast the browser (particularly IE) times out waiting for a response, even though it hasn't finished uploading yet. I would not be surprised if you encounter similar problems. You (and your users) may find it more pleasant to use a upload tool that automatically breaks the file into multiple parts for uploading, and the server re-combines them after the upload is complete. For example, Plupload appears to support this. http://www.plupload.com/ (I've not used plupload, it was just the first hit I got from a search). -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] hacking around
Distracting myself with code diving, I found that internal redirects, implemented by Ns_ConnRedirect and as are used for among other things 404 and 500 error pages, work by changing the url, reauthorizing, and then running the request. However, it does not run preauth and postauth filters. This seems wrong to me, but it seems to have been this way forever. Does anyone have a reason why it shouldn't be changed to run those filters? -J PS: are any other developers even still reading this? -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] hacking around
Ok, it's good that there are other devs reading, it means I can toss out ideas and get useful feedback on them :) Jim Davidson wrote: Howdy, I still watch the list but haven't had time to dig into the code in a long time -- maybe later this summer :) I unfortunately have more time to think about coding on my commute than actually coding. But like you, maybe sometime soon. Anyway, these connection filters are a bit confusing and not well documented -- my fault. Here I think it's ok to not call the filters again but it could go either way. If you update the code, you could make it configurable, default current behavior, to avoid breaking something that assumed the old behavior. In practice I don't think these filters were used very much -- I can only recall the access log modules and one other use in some obscure AOL internal module from years ago. I think there's some OpenACS folks reading at least intermittently, so this may be of interest. I get down into these filters trying to figure out why my OpenACS installation returned error pages for bad urls instead of the configured 404 page. What appears to be going on is that a preauth filter rp_filter is set up to do authentication and some other initialization (importantly, populating ad_conn values), while the main request processor is rp_handler which dispatches the request via a number of different mechanisms and if it fails all of those falls through to ns_returnnotfound, which does the internal 404 redirect. However, the 404 page will also get handled by rp_handler, but based on the values in ad_conn (specifically, extra_url) that are only set up in the preauth filter; if the filter is not run again then it tries to handle the internal 404 url based on the original url and fails again, until it hits the recursion limit and just given an error. (BTW, setting up a 500 error page here crashes the server by overflowing the stack.) This also makes it impossible to use acs-templating on the 404 or 500 error page. Changing the internal redirect logic to call the filters makes this apparently work cleanly in OpenACS. But I could also easily imagine a case where calling those filters twice would badly break things, so making it configurable with the old behavior as the default is probably the best approach. Alternately (if we were to rewind history) is would be useful to set the authorization proc to a tcl callback. There a C api to set it, but that's not exposed at the tcl level. -J -Jim On Jun 2, 2011, at 7:04 PM, Jeff Rogers wrote: Distracting myself with code diving, I found that internal redirects, implemented by Ns_ConnRedirect and as are used for among other things 404 and 500 error pages, work by changing the url, reauthorizing, and then running the request. However, it does not run preauth and postauth filters. This seems wrong to me, but it seems to have been this way forever. Does anyone have a reason why it shouldn't be changed to run those filters? -J PS: are any other developers even still reading this? -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email tolists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email tolists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] hacking around
Don Baccus wrote: On Jun 6, 2011, at 11:31 AM, Jeff Rogers wrote: I get down into these filters trying to figure out why my OpenACS installation returned error pages for bad urls instead of the Yes, this has been a known problem and known to be due to the fact that filter's not being run. So you're saying I could have just asked? :) Well my question for now is, if this was fixed/changed in a way like I've suggested, how long would it be before OpenACS could take advantage of the change? I'm guessing it would need to be at least 2 releases before server requirements could be changed, but that estimate is based on policies I've seen elsewhere, not OpenACS specifically. Are there other aolserver bugs/inadequacies that openacs works around that could be improved? I know that back in the day ArsDigita contributes a number of patches to aolserver (or may have just distributed patches separately; I know that I had a server running 3.1+ad12 at one point). After all, aolserver is a fairly small ecosystem and I believe OpenACS represents a significant part of that, so it makes sense to have some cross-pollination (by which I don't mean adding OpenACS-specific functionality to aolserver, but rather adding/improving general functionality that OpenACS would benefit from). -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] hacking around
Don Baccus wrote: On Jun 6, 2011, at 2:08 PM, Jeff Rogers wrote: Well my question for now is, if this was fixed/changed in a way like I've suggested, how long would it be before OpenACS could take advantage of the change? Wouldn't it just work automagically if the filter were run and context set up properly by rp_filter? Hrm, I guess that's true for what I suggested (modulo a config file addition). I was thinking in the bigger context I suggested tho - if aolserver implemented a new feature that OpenACS could make use of (particularly in core packages), how many releases would need to go by before it could be used. As a contrived example, if authentication (in the sense of Ns_AuthorizeRequest) could be done by a tcl callback, then rp_filter could be a authorization proc instead of a preauth filter, but it would then be incompatible with a version of the server that did not support that. Of course, this is all theoretical right now. It would be a lot easier to reason about it if there was a real use case. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
