Jim, Looks like a lot of really cool stuff.
One question about the ns_quewait: the only events are NS_SOCK_READ and NS_SOCK_WRITE, which matches up with the tcl level, and also match up with Ns_QueueWait. Are the other possible file events handled somewhere else? Note that tcl includes exceptional conditions into read/write, which seems not ideal, but this interface seems to ignore all other conditions. If the idea that the connection will fail by default, which will cause connection abort, then this is a great design. Anyway, for instance, the WaitCallback function distinguishes READ/WRITE/DROP, but the registration proc NsTclQueWaitObjCmd only handles "readable,writable". One huge advantage of the AOLserver "fileevent" interface (over the tcl interface) is that the event type is more clearly defined. This makes callback/handlers a little simpler. My only worry in this particular case is that a connection will get stuck with an unhandled event. We currently define: #define NS_SOCK_READ 0x01 #define NS_SOCK_WRITE 0x02 #define NS_SOCK_EXCEPTION 0x04 #define NS_SOCK_EXIT 0x08 #define NS_SOCK_DROP 0x10 #define NS_SOCK_CANCEL 0x20 #define NS_SOCK_TIMEOUT 0x40 #define NS_SOCK_INIT 0x80 #define NS_SOCK_ANY (NS_SOCK_READ|NS_SOCK_WRITE|NS_SOCK_EXCEPTION) Another question about use of interps. Interps are bound to threads, so they don't move around or follow a connection. The new filter points may create an interp. I'm not sure which thread creates the interp. The prequeue filter runs after all content is uploaded. In the prequeue filter you register a read/write filter (opening a socket). This is quite new, something like a recursive filter. (Or do these filters fire for I/O on the main conn?) Are these interps created and destroyed for each connection, or can they be shared? It seems that there a lot of interesting possibilities with this new code. It is actually difficult to compare with tcl's [fileevent] interface because this appears much more powerful. For instance, it seems very likely that you could turn AOLserver into a proxy server without ever invoking connection threads, everything would be done in high-speed C based event I/O, but the transfer would still have access to a tcl interp. My last question is the initialization of the interp. One driver thread could service multiple virtual servers. When an interp is created for use is there any choice? My understanding of the conn-thread pools is that they partition interps into somewhat similar groups. For instance, thread pools which handle static files would tend to not grow in size over time. Threads which handle adp or tcl files could be expected to grow as they serve unrelated dynamic content. tom jackson On Mon, Dec 7, 2009 at 8:21 PM, Jim Davidson <[email protected]> wrote: > Hi, > > I just checked in some changes to hopefully fix the pre-queue interp leak > muck (and other bugs). I also added read and write filter callbacks -- the > read callbacks can be used to report file upload progress somewhere. And, I > added new ns_cls and ns_quewait commands to work with the curious > Ns_QueueWait interface. Some man pages were updated to reflect the changes > including an example in ns_quewait.n. There's not yet an HTTP protocol > interface on top of ns_quewait but it could be added, letting you do some > REST things, for example, in an event-driven manner before your connection > gets running. > > -Jim > > > On Dec 1, 2009, at 4:45 PM, Jeff Rogers wrote: > >> Jim Davidson wrote: >>> Right -- the pre-queue thing operates within the driver thread only, >>> after all content is read, before it's dispatched to a connection. >>> The idea is that you may want to use the API to fetch using >>> event-style network I/O some other bit of context to attach to the >>> connection using the "connection specific storage" API. So, it won't >>> fire during the read. >> >> Can you share any specific examples of how it has been used? It's always >> struck me as an unfinished (or very specific-purpose) API since it's >> undocumented and it seems that doing anything nontrivial is liable to gum up >> the whole works since the driver is just a single thread. >> >>> However, a progress callback API could be good -- wouldn't be called >>> unless someone wrote something special and careful enough to not bog >>> things down. Maybe an "onstart", "onend", and "on %" and/or "on #" >>> callback which could do something like dump the progress in some >>> shared (process or machine wide) thinger for other threads to check. >>> I like the idea... >>> Oh, and the pre-queue leak sounds like a bug -- should just be >>> re-using the same Tcl interp for all callbacks. >> >> In the case of a tcl filter proc, ProcFilter gets the server/thread specific >> interpreter for the connection and expects NsFreeConnInterp to be called >> later, but in the case of pre-queue filters NsFreeConnInterp is never called >> in the driver thread so it allocates (PopInterp) a new interp every time. >> Adding in a call to NsFreeConnInterp after the prequeue filters looks like >> it fixes the problem. If a filter proc is added into SockRead the same >> thing would need to happen (potentially in the reader thread instead of the >> driver thread). >> >> One thing I am confused about tho, is why without calling NsFreeConnInterp >> in the driver thread it just leaks the interps rather than crashing when it >> tries to use the interp in the conn thread, since it looks like a new conn >> interp wouldn't get allocated in that case. >> >> I also don't understand why there can be multiple interps per server+thread >> combo in the first place (PopInterp/PushInterp); I'd expect that only one >> conn can be in a thread at a time and that it always releases the interp >> when it leaves the thread. >> >> -J >> >>> -Jim >>> On Nov 25, 2009, at 5:46 PM, Jeff Rogers wrote: >>>> It looks like the pre-queue filters are run after the message body >>>> has been read, but before it is passed off to the Conn thread, so >>>> no help there. However it looks like it would not be hard to add >>>> in a new callback to the middle of the read loop, tho it's >>>> debatable if that's a good idea or not (for one, it would get >>>> called a *lot*). >>>> Curious about that tcl prequeue leak. I guess no one uses or cares >>>> about it, since the symptom is serious (more than just a really big >>>> memory leak, it crashes the server too), the cause is pretty >>>> obvious and the fix appears on the surface to be pretty obvious, >>>> although it does result in prequeue filters working differently >>>> from all the others, in particular that it would use a different >>>> interp from the rest of the request. >>>> -J >>>> Tom Jackson wrote: >>>>> There is one possibility. There is a pre-queue filter in >>>>> AOLserver (run inside the driver thread). It works from the Tcl >>>>> level, but creates a memory leak equal to the size of an interp, >>>>> in other words a huge memory leak. However, maybe at the C level, >>>>> you could create a handler which would do something interesting >>>>> before returning control back to the driver thread and ultimately >>>>> the conn thread. I'm not sure exactly when the pre-queue filters >>>>> are activated, but if it is before reading the message body, it >>>>> might be useful. >>>> -- AOLserver - http://www.aolserver.com/ >>>> To Remove yourself from this list, simply send an email to >>>> <[email protected]> with the body of "SIGNOFF AOLSERVER" in >>>> the email message. You can leave the Subject: field of your email >>>> blank. >>> -- AOLserver - http://www.aolserver.com/ >>> To Remove yourself from this list, simply send an email to >>> <[email protected]> with the body of "SIGNOFF AOLSERVER" in >>> the email message. You can leave the Subject: field of your email >>> blank. >> >> >> -- >> AOLserver - http://www.aolserver.com/ >> >> To Remove yourself from this list, simply send an email to >> <[email protected]> with the >> body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: >> field of your email blank. > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > <[email protected]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: > field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[email protected]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
