Re: [AOLSERVER] upload progress
The method of checking progress on a separate URL similar to the example I sent does result in repeated requests during upload. But, they're trivial by comparison - easily in the 100's of req/sec range of response time and throughput. A bit goofy, but over a single keep- alive socket for upload operations (likely rare relative to download, e.g. what you'd expect at YouTube) probably ok Jim Sent from a phone On Jan 19, 2010, at 4:16 PM, Tom Jackson t...@rmadilo.com wrote: This method could also have the advantage of recovery in case of a failed upload. A client would look much like a udp application which tracks packets at the application level. Once the server side API is set, the client could be javascript, java, flash or tcl. The client-side solution also has the advantage of being tailored to the website, and you could use more bandwidth-efficient compression and/or binary transfer instead of encoded transfer. But if you want to provide upload progress to your customers, it seems counterproductive to create a client which queries the progress via a separate url. That just greatly multiplies the number of requests the server must handle. tom jackson On Mon, Jan 18, 2010 at 2:39 AM, John Buckman j...@bookmooch.com wrote: On 11/24/09 5:13 PM, John Buckman wrote: Is there any access (in C or Tcl) to an upload-in-progress in aolserver? It'd be nice if we extended ns_info with [ns_info driver ...] that could give you connection-level info. from the driver thread. In its simplest form, all we need is to expose the total bytes read/written on a socket from the driver thread. Bytes read of the POST request's body and the anticipated Content-Length enables us to compute a rough progress - using the unique URL bit gives us an opaque handle to identify which connection we're interested in. I've learned a few things by deploying a large-file-upload feature on aolserver: 1) IE times out on large file uploads over DSL, as does Chrome and Safari. Only Firefox seems to have a long enough timeout to enable 600mb file uploads over DSL. 2) All the other file upload sites use a client-side widget to upload a file in parts, not using the browser's upload feature at all. Then, they have a thin server-side program which accepts small chunks of the file upload at a time. Once the widget decides the entire file has been sent, it submits to a new web page, which then collects all the small file chunks. So... instead of working on an upload-in-progress feature, it would make sense instead to have a client-side widget (javascript/flash/ java) that sends file upload chunks to a server-side tcl script, and then have a harvester tcl script once the widget says the file upload is complete. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
On Fri, Jan 22, 2010 at 3:03 PM, Jeff Rogers dv...@diphi.com wrote: The YUI upload control looks like a good place to start for the flash client-upload feature. I haven't looked into it too deeply tho, so I don't know what the server side looks like. YUI Uploader widget: http://developer.yahoo.com/yui/uploader/ Other that that, I was pondering the plain upload issue. Since IE/Chrome/Safari are timing out on the upload, I wonder if the connection could be kept alive by sending something - anything - back to the client while it is still uploading. This is just caused by a brain damaged application. TCP/IP handles connection timeouts all by itself. As long as packets are being sent and acknowledged received, the application should not care. But very likely what is happening is that you have a blocking worker thread with is being controlled by another thread just using a simple timeout, without monitoring progress. Anyone who has noticed their browser freeze while loading google analytics, or some other ad iframe has experienced this poor event programming model. Either Firefox avoids this with active monitoring, or it doesn't use a timeout at the application level, or the timeout is very large. This might be doable with Jim's new read filter. Of course, the browsers might respond to data by closing their connection or stopping sending, or crashing (you never know with IE). And then even if it works, you have the problem of not having the tcp connection interrupted for however long it takes, which can be iffy in the world of flaky wireless connections and ISPs. Until the entire POST is complete, you have no method of communicating back to the client, this is the ultimate cause of the no progress being reported. To stay within the HTTP protocol, you would have to send multiple smaller chunks, and wait for the server to acknowledge it has received the data at the application level. Also, the chunked transfer encoding doesn't really help here since proxies are sometimes required to remove this encoding, cache the entire body and maybe retransmit it in chunks. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Ah -- old message I didn't see at first replies in-line below On Dec 9, 2009, at 4:06 PM, Tom Jackson wrote: Jim, Looks like a lot of really cool stuff. One question about the ns_quewait: the only events are NS_SOCK_READ and NS_SOCK_WRITE, which matches up with the tcl level, and also match up with Ns_QueueWait. Are the other possible file events handled somewhere else? Note that tcl includes exceptional conditions into read/write, which seems not ideal, but this interface seems to ignore all other conditions. If the idea that the connection will fail by default, which will cause connection abort, then this is a great design. Anyway, for instance, the WaitCallback function distinguishes READ/WRITE/DROP, but the registration proc NsTclQueWaitObjCmd only handles readable,writable. One huge advantage of the AOLserver fileevent interface (over the tcl interface) is that the event type is more clearly defined. This makes callback/handlers a little simpler. My only worry in this particular case is that a connection will get stuck with an unhandled event. We currently define: #define NS_SOCK_READ 0x01 #define NS_SOCK_WRITE 0x02 #define NS_SOCK_EXCEPTION 0x04 #define NS_SOCK_EXIT 0x08 #define NS_SOCK_DROP 0x10 #define NS_SOCK_CANCEL0x20 #define NS_SOCK_TIMEOUT 0x40 #define NS_SOCK_INIT 0x80 #define NS_SOCK_ANY (NS_SOCK_READ|NS_SOCK_WRITE|NS_SOCK_EXCEPTION) Yup -- at the C level it's read and/or write and the Tcl level read or write. But, the code could be hacked to handle more conditions, e.g., priority data (although I'm not sure what that is) or specific checks for dropped connections (apparently the Ns_Task interface silently sets POLLIN if it sees a POLLHUP but that's not the case elsewhere in the code).Being consistent would be smart although perhaps we may be inviting new bugs in weird ways. Another question about use of interps. Interps are bound to threads, so they don't move around or follow a connection. The new filter points may create an interp. I'm not sure which thread creates the interp. The prequeue filter runs after all content is uploaded. In the prequeue filter you register a read/write filter (opening a socket). This is quite new, something like a recursive filter. (Or do these filters fire for I/O on the main conn?) Are these interps created and destroyed for each connection, or can they be shared? Nope -- the interps are allocated/deallocated as needed just like ordinary connection interps. But, since the connection will shuffle from one thread to the next, interps used by the connection (if any) go through the garbage collection phase as needed, e.g., closing all open files and clearing global vars. This is why the ns_cls interface would be needed to stash per-connection context between threads. And, this means you could have 3 interp/threads involved: -- read callbacks in a reader thread (if configured, optional for ordinary sockets, required for ssl) -- pre-queue callbacks in the driver thread -- normal execution in the connection thread. The code I checked in a month ago tried to deal with all that stuff and avoid the leak that was reported by not doing it properly. Digging in, it was clear the interface wasn't complete -- hopefully it's complete and robust now and the manpages are close to accurate. It seems that there a lot of interesting possibilities with this new code. It is actually difficult to compare with tcl's [fileevent] interface because this appears much more powerful. For instance, it seems very likely that you could turn AOLserver into a proxy server without ever invoking connection threads, everything would be done in high-speed C based event I/O, but the transfer would still have access to a tcl interp. Yup -- the interface is a bit arcane but it can do interesting things like that. My last question is the initialization of the interp. One driver thread could service multiple virtual servers. When an interp is created for use is there any choice? My understanding of the conn-thread pools is that they partition interps into somewhat similar groups. For instance, thread pools which handle static files would tend to not grow in size over time. Threads which handle adp or tcl files could be expected to grow as they serve unrelated dynamic content. The interps are allocated from per-server caches just like in a connection thread so the state should look like you expect (although, as mentioned, global vars will disappear between the reader/pre-queue interps and the connection interps). As the driver thread never exists, misused interps in this case would lead to memory leaks/bloat that may be possible to mitigate in connection threads via the die after so many connections... options. I suppose we could add a die after so many uses... config to get the same
Re: [AOLSERVER] upload progress
The YUI upload control looks like a good place to start for the flash client-upload feature. I haven't looked into it too deeply tho, so I don't know what the server side looks like. YUI Uploader widget: http://developer.yahoo.com/yui/uploader/ Other that that, I was pondering the plain upload issue. Since IE/Chrome/Safari are timing out on the upload, I wonder if the connection could be kept alive by sending something - anything - back to the client while it is still uploading. This might be doable with Jim's new read filter. Of course, the browsers might respond to data by closing their connection or stopping sending, or crashing (you never know with IE). And then even if it works, you have the problem of not having the tcp connection interrupted for however long it takes, which can be iffy in the world of flaky wireless connections and ISPs. -J Tom Jackson wrote: This method could also have the advantage of recovery in case of a failed upload. A client would look much like a udp application which tracks packets at the application level. Once the server side API is set, the client could be javascript, java, flash or tcl. The client-side solution also has the advantage of being tailored to the website, and you could use more bandwidth-efficient compression and/or binary transfer instead of encoded transfer. But if you want to provide upload progress to your customers, it seems counterproductive to create a client which queries the progress via a separate url. That just greatly multiplies the number of requests the server must handle. tom jackson On Mon, Jan 18, 2010 at 2:39 AM, John Buckman j...@bookmooch.com wrote: On 11/24/09 5:13 PM, John Buckman wrote: Is there any access (in C or Tcl) to an upload-in-progress in aolserver? It'd be nice if we extended ns_info with [ns_info driver ...] that could give you connection-level info. from the driver thread. In its simplest form, all we need is to expose the total bytes read/written on a socket from the driver thread. Bytes read of the POST request's body and the anticipated Content-Length enables us to compute a rough progress - using the unique URL bit gives us an opaque handle to identify which connection we're interested in. I've learned a few things by deploying a large-file-upload feature on aolserver: 1) IE times out on large file uploads over DSL, as does Chrome and Safari. Only Firefox seems to have a long enough timeout to enable 600mb file uploads over DSL. 2) All the other file upload sites use a client-side widget to upload a file in parts, not using the browser's upload feature at all. Then, they have a thin server-side program which accepts small chunks of the file upload at a time. Once the widget decides the entire file has been sent, it submits to a new web page, which then collects all the small file chunks. So... instead of working on an upload-in-progress feature, it would make sense instead to have a client-side widget (javascript/flash/java) that sends file upload chunks to a server-side tcl script, and then have a harvester tcl script once the widget says the file upload is complete. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
I don't have any problem with this solution. It is superior to using a
forward proxy which uploads the entire file then reports progress to
the final server (this was the original model proposed in this thread,
by example).
In fact, I pointed out that the server thread is a proxy, handling
upload prior to allocating a conn thread. If you could peek into this
process, you get feedback.
But this is extremely inefficient. All of these solutions require a
specialized client, even it it seems somewhat transparent to the end
user. So I have pointed out several times that the best solution is a
client side solution which tracks upload progress by knowing the total
file size and the amount of bytes sent.
This really only make sense when the client is somewhat smaller than
the typical upload. Also, downloads are usually faster than uploads,
so the specialized client looks more attractive. the only critical
factor is ease of installation of the client.
Given the size of a tcl client, about 2 meg, any website with a
typical upload of 3+ megs would benefit from an easy to install and
use specialized client. My guess is that the javascript, flash and
java clients could be smaller, but would vary more than a simple tcl
client, which would work unchanged at the script level.
tom jackson
On Mon, Jan 18, 2010 at 12:51 PM, Jim Davidson jgdavid...@mac.com wrote:
Hi,
I think we were talking about this about a month ago. I updated the source
to enable upload-progress checking with a combination of ns_register_filter
and nsv -- there's an example at the latest ns_register_filter man page
(pasted below). This may work for you although it would require compiling
from latest sources. It assumes you have some javascript thinger that makes
repeated calls to check the status of the upload in progress on another
thread.
-Jim
EXAMPLE
The following example uses a read filter to update status of
a large HTTP POST to the
/upload/key url where key is some client-specified unique value.
While the upload is in
progress, it can be monitored with repeated GET requests to the
/status/key url with the same
key:
#
# Register procs to receive uploads and check status
# mainted in an nsv array.
#
ns_register_proc POST /upload upload.post
ns_register_proc GET /status upload.status
proc upload.status {} {
set key [ns_conn urlv 1]
if {[catch {set status [nsv_get status $key]}]} {
set status unknown
}
ns_return 200 text/plain $status
}
proc upload.post {} {
set key [ns_conn urlv 1]
nsv_unset status $key
# ... do something with content ...
ns_return 200 text/plain received
}
#
# Register a read filter ot update status
#
ns_register_filter read POST /upload/* upload.update
proc upload.update {why} {
set key [ns_conn urlv 1]
set expected [ns_conn contentlength]
set received [ns_conn contentavail]
set status [list $expected $received]
nsv_set status $key $status
return filter_ok
}
On Jan 18, 2010, at 2:39 AM, John Buckman wrote:
On 11/24/09 5:13 PM, John Buckman wrote:
Is there any access (in C or Tcl) to an upload-in-progress in aolserver?
It'd be nice if we extended ns_info with [ns_info driver ...] that could
give you connection-level info. from the driver thread. In its simplest
form, all we need is to expose the total bytes read/written on a socket
from the driver thread. Bytes read of the POST request's body and the
anticipated Content-Length enables us to compute a rough progress -
using the unique URL bit gives us an opaque handle to identify which
connection we're interested in.
I've learned a few things by deploying a large-file-upload feature on
aolserver:
1) IE times out on large file uploads over DSL, as does Chrome and Safari.
Only Firefox seems to have a long enough timeout to enable 600mb file
uploads over DSL.
2) All the other file upload sites use a client-side widget to upload a file
in parts, not using the browser's upload feature at all. Then, they have a
thin server-side program which accepts small chunks of the file upload at a
time. Once the widget decides the entire file has been sent, it submits to a
new web page, which then collects all the small file chunks.
So... instead of working on an upload-in-progress feature, it would make
sense instead to have a client-side widget (javascript/flash/java) that
sends file upload chunks to a server-side tcl script, and then have a
harvester tcl script once the widget says the file upload
Re: [AOLSERVER] upload progress
Hi,
I think we were talking about this about a month ago. I updated the source to
enable upload-progress checking with a combination of ns_register_filter and
nsv -- there's an example at the latest ns_register_filter man page (pasted
below). This may work for you although it would require compiling from latest
sources. It assumes you have some javascript thinger that makes repeated calls
to check the status of the upload in progress on another thread.
-Jim
EXAMPLE
The following example uses a read filter to update status of
a large HTTP POST to the
/upload/key url where key is some client-specified unique value.
While the upload is in
progress, it can be monitored with repeated GET requests to the
/status/key url with the same
key:
#
# Register procs to receive uploads and check status
# mainted in an nsv array.
#
ns_register_proc POST /upload upload.post
ns_register_proc GET /status upload.status
proc upload.status {} {
set key [ns_conn urlv 1]
if {[catch {set status [nsv_get status $key]}]} {
set status unknown
}
ns_return 200 text/plain $status
}
proc upload.post {} {
set key [ns_conn urlv 1]
nsv_unset status $key
# ... do something with content ...
ns_return 200 text/plain received
}
#
# Register a read filter ot update status
#
ns_register_filter read POST /upload/* upload.update
proc upload.update {why} {
set key [ns_conn urlv 1]
set expected [ns_conn contentlength]
set received [ns_conn contentavail]
set status [list $expected $received]
nsv_set status $key $status
return filter_ok
}
On Jan 18, 2010, at 2:39 AM, John Buckman wrote:
On 11/24/09 5:13 PM, John Buckman wrote:
Is there any access (in C or Tcl) to an upload-in-progress in aolserver?
It'd be nice if we extended ns_info with [ns_info driver ...] that could
give you connection-level info. from the driver thread. In its simplest
form, all we need is to expose the total bytes read/written on a socket
from the driver thread. Bytes read of the POST request's body and the
anticipated Content-Length enables us to compute a rough progress -
using the unique URL bit gives us an opaque handle to identify which
connection we're interested in.
I've learned a few things by deploying a large-file-upload feature on
aolserver:
1) IE times out on large file uploads over DSL, as does Chrome and Safari.
Only Firefox seems to have a long enough timeout to enable 600mb file uploads
over DSL.
2) All the other file upload sites use a client-side widget to upload a file
in parts, not using the browser's upload feature at all. Then, they have a
thin server-side program which accepts small chunks of the file upload at a
time. Once the widget decides the entire file has been sent, it submits to a
new web page, which then collects all the small file chunks.
So... instead of working on an upload-in-progress feature, it would make
sense instead to have a client-side widget (javascript/flash/java) that sends
file upload chunks to a server-side tcl script, and then have a harvester
tcl script once the widget says the file upload is complete.
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] upload progress
On 11/24/09 5:13 PM, John Buckman wrote: Is there any access (in C or Tcl) to an upload-in-progress in aolserver? It'd be nice if we extended ns_info with [ns_info driver ...] that could give you connection-level info. from the driver thread. In its simplest form, all we need is to expose the total bytes read/written on a socket from the driver thread. Bytes read of the POST request's body and the anticipated Content-Length enables us to compute a rough progress - using the unique URL bit gives us an opaque handle to identify which connection we're interested in. Fortunately for me, I haven't built any applications where large file upload handling has been a requirement. ;-) -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Hi, I just checked in some changes to hopefully fix the pre-queue interp leak muck (and other bugs). I also added read and write filter callbacks -- the read callbacks can be used to report file upload progress somewhere. And, I added new ns_cls and ns_quewait commands to work with the curious Ns_QueueWait interface. Some man pages were updated to reflect the changes including an example in ns_quewait.n. There's not yet an HTTP protocol interface on top of ns_quewait but it could be added, letting you do some REST things, for example, in an event-driven manner before your connection gets running. -Jim On Dec 1, 2009, at 4:45 PM, Jeff Rogers wrote: Jim Davidson wrote: Right -- the pre-queue thing operates within the driver thread only, after all content is read, before it's dispatched to a connection. The idea is that you may want to use the API to fetch using event-style network I/O some other bit of context to attach to the connection using the connection specific storage API. So, it won't fire during the read. Can you share any specific examples of how it has been used? It's always struck me as an unfinished (or very specific-purpose) API since it's undocumented and it seems that doing anything nontrivial is liable to gum up the whole works since the driver is just a single thread. However, a progress callback API could be good -- wouldn't be called unless someone wrote something special and careful enough to not bog things down. Maybe an onstart, onend, and on % and/or on # callback which could do something like dump the progress in some shared (process or machine wide) thinger for other threads to check. I like the idea... Oh, and the pre-queue leak sounds like a bug -- should just be re-using the same Tcl interp for all callbacks. In the case of a tcl filter proc, ProcFilter gets the server/thread specific interpreter for the connection and expects NsFreeConnInterp to be called later, but in the case of pre-queue filters NsFreeConnInterp is never called in the driver thread so it allocates (PopInterp) a new interp every time. Adding in a call to NsFreeConnInterp after the prequeue filters looks like it fixes the problem. If a filter proc is added into SockRead the same thing would need to happen (potentially in the reader thread instead of the driver thread). One thing I am confused about tho, is why without calling NsFreeConnInterp in the driver thread it just leaks the interps rather than crashing when it tries to use the interp in the conn thread, since it looks like a new conn interp wouldn't get allocated in that case. I also don't understand why there can be multiple interps per server+thread combo in the first place (PopInterp/PushInterp); I'd expect that only one conn can be in a thread at a time and that it always releases the interp when it leaves the thread. -J -Jim On Nov 25, 2009, at 5:46 PM, Jeff Rogers wrote: It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty obvious, although it does result in prequeue filters working differently from all the others, in particular that it would use a different interp from the rest of the request. -J Tom Jackson wrote: There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can
Re: [AOLSERVER] upload progress
Right -- the pre-queue thing operates within the driver thread only, after all content is read, before it's dispatched to a connection. The idea is that you may want to use the API to fetch using event-style network I/O some other bit of context to attach to the connection using the connection specific storage API. So, it won't fire during the read. However, a progress callback API could be good -- wouldn't be called unless someone wrote something special and careful enough to not bog things down. Maybe an onstart, onend, and on % and/or on # callback which could do something like dump the progress in some shared (process or machine wide) thinger for other threads to check. I like the idea... Oh, and the pre-queue leak sounds like a bug -- should just be re-using the same Tcl interp for all callbacks. -Jim On Nov 25, 2009, at 5:46 PM, Jeff Rogers wrote: It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty obvious, although it does result in prequeue filters working differently from all the others, in particular that it would use a different interp from the rest of the request. -J Tom Jackson wrote: There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Jim Davidson wrote: Right -- the pre-queue thing operates within the driver thread only, after all content is read, before it's dispatched to a connection. The idea is that you may want to use the API to fetch using event-style network I/O some other bit of context to attach to the connection using the connection specific storage API. So, it won't fire during the read. Can you share any specific examples of how it has been used? It's always struck me as an unfinished (or very specific-purpose) API since it's undocumented and it seems that doing anything nontrivial is liable to gum up the whole works since the driver is just a single thread. However, a progress callback API could be good -- wouldn't be called unless someone wrote something special and careful enough to not bog things down. Maybe an onstart, onend, and on % and/or on # callback which could do something like dump the progress in some shared (process or machine wide) thinger for other threads to check. I like the idea... Oh, and the pre-queue leak sounds like a bug -- should just be re-using the same Tcl interp for all callbacks. In the case of a tcl filter proc, ProcFilter gets the server/thread specific interpreter for the connection and expects NsFreeConnInterp to be called later, but in the case of pre-queue filters NsFreeConnInterp is never called in the driver thread so it allocates (PopInterp) a new interp every time. Adding in a call to NsFreeConnInterp after the prequeue filters looks like it fixes the problem. If a filter proc is added into SockRead the same thing would need to happen (potentially in the reader thread instead of the driver thread). One thing I am confused about tho, is why without calling NsFreeConnInterp in the driver thread it just leaks the interps rather than crashing when it tries to use the interp in the conn thread, since it looks like a new conn interp wouldn't get allocated in that case. I also don't understand why there can be multiple interps per server+thread combo in the first place (PopInterp/PushInterp); I'd expect that only one conn can be in a thread at a time and that it always releases the interp when it leaves the thread. -J -Jim On Nov 25, 2009, at 5:46 PM, Jeff Rogers wrote: It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty obvious, although it does result in prequeue filters working differently from all the others, in particular that it would use a different interp from the rest of the request. -J Tom Jackson wrote: There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
On Dec 1, 2009, at 4:45 PM, Jeff Rogers wrote: Jim Davidson wrote: Right -- the pre-queue thing operates within the driver thread only, after all content is read, before it's dispatched to a connection. The idea is that you may want to use the API to fetch using event-style network I/O some other bit of context to attach to the connection using the connection specific storage API. So, it won't fire during the read. Can you share any specific examples of how it has been used? It's always struck me as an unfinished (or very specific-purpose) API since it's undocumented and it seems that doing anything nontrivial is liable to gum up the whole works since the driver is just a single thread. Nope, no examples I'm aware of. You're right it's unfinished, mostly because it's missing a curl-like simple HTTP interface on top and a Tcl interface to configure it to do the common thing of fetching something over the net via REST or SOAP and shoving the result into CLS for later use. And, there's no documentation so how would anyone know what or why it would be useful? Also, it's a bit advanced as it were for just the reason you mentioned -- if you mucked up your callback, you would stall all accept and read-ahead. Frankly, I always thought it was a cool idea but could never get anyone else interested. In practice folks had various search database here... or fetch xml there... type stuff riddled throughout their existing ADP scripts and it wasn't strictly necessary or a priority to re-factor that stuff to move it all to the pre-queue interface. My goal of getting all the context into the connection before processing began, eliminating any potential long-waiting stalls in the connection threads was never something that got others super excited. Even so, the concept is still cool. However, a progress callback API could be good -- wouldn't be called unless someone wrote something special and careful enough to not bog things down. Maybe an onstart, onend, and on % and/or on # callback which could do something like dump the progress in some shared (process or machine wide) thinger for other threads to check. I like the idea... Oh, and the pre-queue leak sounds like a bug -- should just be re-using the same Tcl interp for all callbacks. In the case of a tcl filter proc, ProcFilter gets the server/thread specific interpreter for the connection and expects NsFreeConnInterp to be called later, but in the case of pre-queue filters NsFreeConnInterp is never called in the driver thread so it allocates (PopInterp) a new interp every time. Adding in a call to NsFreeConnInterp after the prequeue filters looks like it fixes the problem. If a filter proc is added into SockRead the same thing would need to happen (potentially in the reader thread instead of the driver thread). Ah ... this does sound like a bug. Since the interface is undocumented and not used, I don't think it would hurt to add a call to NsFreeConnInterp as a special case of a pre-queue filter. NsFreeConnInterp does call a free conn Tcl trace which assumes end of connection but I guess that's ok to call pre-queue (if there has ever been one registered). One thing I am confused about tho, is why without calling NsFreeConnInterp in the driver thread it just leaks the interps rather than crashing when it tries to use the interp in the conn thread, since it looks like a new conn interp wouldn't get allocated in that case. Hmm... appears it's working when it shouldn't, Interps are supposed to be per-thread but maybe in this case the interp is floating from one thread to the next and still working, avoiding things that wouldn't work. Anyway, calling NsFreeConnInterp should clear this out. I also don't understand why there can be multiple interps per server+thread combo in the first place (PopInterp/PushInterp); I'd expect that only one conn can be in a thread at a time and that it always releases the interp when it leaves the thread. I think there was an edge case that led to a need for a cache of possible interps instead of just holding one. In practice, it's always just one unless someone writes some weird code to do a pop/push directly to have some alternate interp for a special purpose. -Jim -J -Jim On Nov 25, 2009, at 5:46 PM, Jeff Rogers wrote: It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty
Re: [AOLSERVER] upload progress
Jeff, Interps are confined to a specific thread. You can transfer the sock around, but not the interp. But the big reason for different interps is that they are or can be specialized. The prequeue interp could be very simple. Conn interps tend to be big and expensive so you don't want to use them any more than you have to. This is why we have a driver thread handling all i/o prior to queuing the conn. The driver thread is the upload equivalent of the download helper. (plus a bunch of other stuff). As an example of an additional prequeue filter, maybe the new http client (written in C) using the ?ns_task? API. Basically seems like you could dole out a task and return once the task (getting something via event driven http) is complete. Maybe a good use of the prequeue filter would be to actually return this download progress information. Since it is a filter, it would fire on a particular url. Maybe you could do a quick return and abort the connection, never using a conn thread. (Note this is a separate http request from the upload request.) It should be very fast if you never fire up an interp. tom jackson On Tue, Dec 1, 2009 at 1:45 PM, Jeff Rogers dv...@diphi.com wrote: Jim Davidson wrote: Right -- the pre-queue thing operates within the driver thread only, after all content is read, before it's dispatched to a connection. The idea is that you may want to use the API to fetch using event-style network I/O some other bit of context to attach to the connection using the connection specific storage API. So, it won't fire during the read. Can you share any specific examples of how it has been used? It's always struck me as an unfinished (or very specific-purpose) API since it's undocumented and it seems that doing anything nontrivial is liable to gum up the whole works since the driver is just a single thread. However, a progress callback API could be good -- wouldn't be called unless someone wrote something special and careful enough to not bog things down. Maybe an onstart, onend, and on % and/or on # callback which could do something like dump the progress in some shared (process or machine wide) thinger for other threads to check. I like the idea... Oh, and the pre-queue leak sounds like a bug -- should just be re-using the same Tcl interp for all callbacks. In the case of a tcl filter proc, ProcFilter gets the server/thread specific interpreter for the connection and expects NsFreeConnInterp to be called later, but in the case of pre-queue filters NsFreeConnInterp is never called in the driver thread so it allocates (PopInterp) a new interp every time. Adding in a call to NsFreeConnInterp after the prequeue filters looks like it fixes the problem. If a filter proc is added into SockRead the same thing would need to happen (potentially in the reader thread instead of the driver thread). One thing I am confused about tho, is why without calling NsFreeConnInterp in the driver thread it just leaks the interps rather than crashing when it tries to use the interp in the conn thread, since it looks like a new conn interp wouldn't get allocated in that case. I also don't understand why there can be multiple interps per server+thread combo in the first place (PopInterp/PushInterp); I'd expect that only one conn can be in a thread at a time and that it always releases the interp when it leaves the thread. -J -Jim On Nov 25, 2009, at 5:46 PM, Jeff Rogers wrote: It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty obvious, although it does result in prequeue filters working differently from all the others, in particular that it would use a different interp from the rest of the request. -J Tom Jackson wrote: There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the
Re: [AOLSERVER] upload progress
On Tue, Dec 1, 2009 at 3:28 PM, Jim Davidson jgdavid...@mac.com wrote: On Dec 1, 2009, at 4:45 PM, Jeff Rogers wrote: I also don't understand why there can be multiple interps per server+thread combo in the first place (PopInterp/PushInterp); I'd expect that only one conn can be in a thread at a time and that it always releases the interp when it leaves the thread. I think there was an edge case that led to a need for a cache of possible interps instead of just holding one. In practice, it's always just one unless someone writes some weird code to do a pop/push directly to have some alternate interp for a special purpose. One thing which causes multiple interps per thread is the default threadpool. If you have a default threadpool the threads are shared across servers. So, as each server uses a thread, an interp for that server is created. My advice is to always register a default threadpool for each server so that the process wide default threadpool is never used. I'm not sure if there is any security problem, but a shared threadpool makes each thread more expensive, and I doubt it saves resources. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
John, I'm just going to venture a guess. I hope that Jim D. or someone else more familiar with the internals will set me straight. The problem with upload progress monitoring is that uploads are finished before a conn thread is allocated. Uploads are done in the driver thread, or a worker thread before the conn thread is given control. One thing which could help is if AOLserver introduced chunked transfer encoding for uploads. But to report progress back to the user would require a different setup. You would need to monitor the progress of some other process/thread. AOLserver has chosen to stratify io. By stratify I mean the io handling is moved through a pipeline. The pipeline may transfer handling between threads but also enforces the direction of communication. There is no asynchronous io, each step in the pipeline is focused on one-way communication. For instance, the driver thread only reads, the conn threads only write. There is a kind of fiction that conn threads do any reading from the client. These threads only read from buffers or disk. The reason should be obvious: dos attacks. The driver thread is highly tuned to protect the expensive conn threads. In addition to that issue, the connio uses scatter/gather io, so the application gets a very imperfect view of the actual progress. Of course the io is much more efficient and progress reporting would require an independent observation and messaging system. A first step would be the ability to log upload progress to error.log. If you can't do that, reporting back to the client will be impossible (unless you think the logging system should operate with less available information than individual conn threads). There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. tom jackson On Tue, Nov 24, 2009 at 2:13 PM, John Buckman j...@bookmooch.com wrote: Naviserver has a very nice feature that allows (via javascript) to show a user the percent upload progress of a file. I tried porting their progress.c file to aolserver, but it's a significant effort, as it depends on other changes naviserver has implemented to the aolserver code. However, I was wondering if there wasn't a fairly simple way to implement something similar on aolserver. What I'd need is a way to know how much data has been uploaded. The way naviserver does it, is by asking you to POST your upload to a unique URL, and then they provide a tcl command that returns how many bytes have been uploaded to that unique URL. A javascript regularly polls an ADP page that uses the tcl progress command to tell you the number of bytes uploaded. Is there any access (in C or Tcl) to an upload-in-progress in aolserver? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Tom Jackson schrieb: John, I'm just going to venture a guess. I hope that Jim D. or someone else more familiar with the internals will set me straight. The problem with upload progress monitoring is that uploads are finished before a conn thread is allocated. Uploads are done in the driver thread, or a worker thread before the conn thread is given control. Tom, the typical meachnism for upload progress bars is that separate (ajax) queries are used to query the state of an running upload, which is identified by some unique ID (e.g. X-Progress-ID in http://wiki.nginx.org/NginxHttpUploadProgressModule, or some other heuristics. e.g. based on URL and peer addr). So, one needs just a thread-safe introspection meachnism, which checks in the spooler the state (such as ns_upload_stats in naviserver) and returns it back. The spooler hast just to update these statistics when it recieves data. -gustaf neumann -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
the typical meachnism for upload progress bars is that separate (ajax) queries are used to query the state of an running upload, which is identified by some unique ID (e.g. X-Progress-ID in http://wiki.nginx.org/NginxHttpUploadProgressModule, or some other heuristics. e.g. based on URL and peer addr). So, one needs just a thread-safe introspection meachnism, which checks in the spooler the state (such as ns_upload_stats in naviserver) and returns it back. The spooler hast just to update these statistics when it recieves data. Exactly - what is needed in aolserver is simply a tcl command for introspection, specifically how many bytes have been received in some particular case. That data must be somewhere, what I was wondering is if anyone had an idea *where*. -j -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Gustaf, I've seen these working, although I'm never sure where exactly the magic happens. It looks like the ngix idea is to work as a proxy: It works because Nginx acts as an accelerator of an upstream server, storing uploaded POST content on disk, before transmitting it to the upstream server. Each individual POST upload request should contain a progress unique identifier. I wonder if the progress is from you to ngix, or from ngix to the final server. On Wed, Nov 25, 2009 at 4:07 AM, Gustaf Neumann neum...@wu.ac.at wrote: Tom Jackson schrieb: John, I'm just going to venture a guess. I hope that Jim D. or someone else more familiar with the internals will set me straight. The problem with upload progress monitoring is that uploads are finished before a conn thread is allocated. Uploads are done in the driver thread, or a worker thread before the conn thread is given control. Tom, the typical meachnism for upload progress bars is that separate (ajax) queries are used to query the state of an running upload, which is identified by some unique ID (e.g. X-Progress-ID in http://wiki.nginx.org/NginxHttpUploadProgressModule, or some other heuristics. e.g. based on URL and peer addr). So, one needs just a thread-safe introspection meachnism, which checks in the spooler the state (such as ns_upload_stats in naviserver) and returns it back. The spooler hast just to update these statistics when it recieves data. -gustaf neumann -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Gustaf, Oops, accidentally hit send. I just started work on an event driven http client (called htclient). It can monitor downloads just by using a variable trace. I haven't reversed the idea for uploads yet, but it would be easy. Not so easy is guessing the length of the encoded file prior to sending. Seems like a better solution for large file uploads would be to do the upload as binary data instead of multi-part and get direct local monitoring and the ability to cancel, and/or restart a failed upload. For clients who have tcl/tk installed, it would be easy to auto-generate a tcl script to handle one or several uploads. tom jackson On Wed, Nov 25, 2009 at 9:13 AM, Tom Jackson t...@rmadilo.com wrote: Gustaf, I've seen these working, although I'm never sure where exactly the magic happens. It looks like the ngix idea is to work as a proxy: It works because Nginx acts as an accelerator of an upstream server, storing uploaded POST content on disk, before transmitting it to the upstream server. Each individual POST upload request should contain a progress unique identifier. I wonder if the progress is from you to ngix, or from ngix to the final server. On Wed, Nov 25, 2009 at 4:07 AM, Gustaf Neumann neum...@wu.ac.at wrote: Tom Jackson schrieb: John, I'm just going to venture a guess. I hope that Jim D. or someone else more familiar with the internals will set me straight. The problem with upload progress monitoring is that uploads are finished before a conn thread is allocated. Uploads are done in the driver thread, or a worker thread before the conn thread is given control. Tom, the typical meachnism for upload progress bars is that separate (ajax) queries are used to query the state of an running upload, which is identified by some unique ID (e.g. X-Progress-ID in http://wiki.nginx.org/NginxHttpUploadProgressModule, or some other heuristics. e.g. based on URL and peer addr). So, one needs just a thread-safe introspection meachnism, which checks in the spooler the state (such as ns_upload_stats in naviserver) and returns it back. The spooler hast just to update these statistics when it recieves data. -gustaf neumann -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty obvious, although it does result in prequeue filters working differently from all the others, in particular that it would use a different interp from the rest of the request. -J Tom Jackson wrote: There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
