Re: [AOLSERVER] nsdci query
On 10/30/07, Damien O'Rourke <[EMAIL PROTECTED]> wrote: > Jim, > > thanks for the info. > Keeping it all in utf-8 makes sense as you say ... so long as you > don't need to access binary files directly with another app. If all > send/receive requests go through the sob it works fine ... I have one > service using it that way at the moment. Debating about whether to > switch it to using nproxy ... there's some sense in making it the > gateway for all requests for those files ... there are other systems > that use similar models e.g. email de-duping software i.e. the black > box service ... who knows how YouTube or AWS actually store the > data ... could be own propriety format for all I can know ... all I > see is what they present to me when I request an item. > > Damien See Google Tech Talk video on YouTube Scalability http://kylecordes.com/2007/07/12/youtube-scalability/ --tkosiak -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Jim, thanks for the info. Keeping it all in utf-8 makes sense as you say ... so long as you don't need to access binary files directly with another app. If all send/receive requests go through the sob it works fine ... I have one service using it that way at the moment. Debating about whether to switch it to using nproxy ... there's some sense in making it the gateway for all requests for those files ... there are other systems that use similar models e.g. email de-duping software i.e. the black box service ... who knows how YouTube or AWS actually store the data ... could be own propriety format for all I can know ... all I see is what they present to me when I request an item. Damien On 28/10/2007, at 5:35 AM, Jim Davidson wrote: Howdy, I wrote the core SOB code years ago -- at the time I think our goals included: -- Security: Server connects out to known clients. -- Performance: State-full connections, minimal protocol overhead, lots of caching, no encoding overhead -- Simple: String keys, string data, integration with net cache flush -- Insight: Lot's of traffic counters collected -- Tcl aware: Expects to store/send UTF-8 strings only -- Some special case stuff, e.g., the comment board code. In general, SOB met these goals at the expense of: -- Config complexity: Maintaining topology is rationale but tedious compared to client-server HTTP connections through routers -- Binary: Just strings -- File system scale: SOB relies on rationale topology and/or key management to avoid file system scale problems (i.e., too many files in a directory). The point on UTF-8 deserves a bit more explanation. In general, all the parts of the "dci" module assumed everything is in UTF-8, always. In a sense, it's best to imagine SOB (and NV, AV, etc.) as really just a way to serialize Tcl data to/from some stable something -- if it's not in memory, it's designed to be read directly into memory with no encoding / translation / etc. In other words, the lack of encoding and binary code support was by design, not an oversight, which provided some benefits but -- as in the case of trying to use SOB for thumbnails -- some downside. -Jim On Oct 25, 2007, at 9:12 AM, Michael Andrews wrote: In general, I find the SOB model works well. The caching on the server and the client provide unmatched performance. The NCF provides a mechanism to flush the cache on updates. The Devil is always in the details. How you set up multiple SOB servers and then get the publishes to multiple SOB servers can certainly create scaling and topology management issues. Most sites to do not have the demands that AOL has - and a simple SOB solution would be nice addition to any content distribution system. However there are advantages to stateless systems. :-) (How's that for diplomatic) M On Oct 25, 2007, at 10:19 AM, Dossy Shiobara wrote: On 2007.10.25, Jay Rohr <[EMAIL PROTECTED]> wrote: In general, I would not recommend using sob. It was developed for very specific requirements at AOL and it does not scale well under load - volume or frequency. We are in the process of replacing it with something else. Okay, this is just plain misinformation. SOB works very well for what it was intended for, a low-write, high-read, distributed service for small objects (thus, the name "SOB"). Any scaling issues AOL is having has to do with how it has since been misused and/or operationalized. -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Howdy, I wrote the core SOB code years ago -- at the time I think our goals included: -- Security: Server connects out to known clients. -- Performance: State-full connections, minimal protocol overhead, lots of caching, no encoding overhead -- Simple: String keys, string data, integration with net cache flush -- Insight: Lot's of traffic counters collected -- Tcl aware: Expects to store/send UTF-8 strings only -- Some special case stuff, e.g., the comment board code. In general, SOB met these goals at the expense of: -- Config complexity: Maintaining topology is rationale but tedious compared to client-server HTTP connections through routers -- Binary: Just strings -- File system scale: SOB relies on rationale topology and/or key management to avoid file system scale problems (i.e., too many files in a directory). The point on UTF-8 deserves a bit more explanation. In general, all the parts of the "dci" module assumed everything is in UTF-8, always. In a sense, it's best to imagine SOB (and NV, AV, etc.) as really just a way to serialize Tcl data to/from some stable something -- if it's not in memory, it's designed to be read directly into memory with no encoding / translation / etc. In other words, the lack of encoding and binary code support was by design, not an oversight, which provided some benefits but -- as in the case of trying to use SOB for thumbnails -- some downside. -Jim On Oct 25, 2007, at 9:12 AM, Michael Andrews wrote: In general, I find the SOB model works well. The caching on the server and the client provide unmatched performance. The NCF provides a mechanism to flush the cache on updates. The Devil is always in the details. How you set up multiple SOB servers and then get the publishes to multiple SOB servers can certainly create scaling and topology management issues. Most sites to do not have the demands that AOL has - and a simple SOB solution would be nice addition to any content distribution system. However there are advantages to stateless systems. :-) (How's that for diplomatic) M On Oct 25, 2007, at 10:19 AM, Dossy Shiobara wrote: On 2007.10.25, Jay Rohr <[EMAIL PROTECTED]> wrote: In general, I would not recommend using sob. It was developed for very specific requirements at AOL and it does not scale well under load - volume or frequency. We are in the process of replacing it with something else. Okay, this is just plain misinformation. SOB works very well for what it was intended for, a low-write, high-read, distributed service for small objects (thus, the name "SOB"). Any scaling issues AOL is having has to do with how it has since been misused and/or operationalized. -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
On 2007.10.26, Damien O'Rourke <[EMAIL PROTECTED]> wrote: > It was the two tier caching for performance and the auto flush that > appealed to me ... serving lots of thumbnail images which occasionally > get updated, and over time get outdated and unused, being replaced in > use, and cache, by others. > > Pity then sob doesn't do binary ... Enhancing SOB to properly round-trip binary data isn't unreasonably difficult. Maybe I'll tackle that after I get tired of hacking on nsjsapi ... -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
It was the two tier caching for performance and the auto flush that appealed to me ... serving lots of thumbnail images which occasionally get updated, and over time get outdated and unused, being replaced in use, and cache, by others. Pity then sob doesn't do binary ... Any advice from experience on nproxy in relation to scaling and load Damien -Original Message- From: AOLserver Discussion on behalf of Michael Andrews Sent: Fri 10/26/2007 2:12 AM To: AOLSERVER@LISTSERV.AOL.COM Subject: Re: [AOLSERVER] nsdci query In general, I find the SOB model works well. The caching on the server and the client provide unmatched performance. The NCF provides a mechanism to flush the cache on updates. The Devil is always in the details. How you set up multiple SOB servers and then get the publishes to multiple SOB servers can certainly create scaling and topology management issues. Most sites to do not have the demands that AOL has - and a simple SOB solution would be nice addition to any content distribution system. However there are advantages to stateless systems. :-) (How's that for diplomatic) M On Oct 25, 2007, at 10:19 AM, Dossy Shiobara wrote: > On 2007.10.25, Jay Rohr <[EMAIL PROTECTED]> wrote: >> In general, I would not recommend using sob. It was developed for >> very >> specific requirements at AOL and it does not scale well under load - >> volume or frequency. We are in the process of replacing it with >> something else. > > Okay, this is just plain misinformation. SOB works very well for what > it was intended for, a low-write, high-read, distributed service for > small objects (thus, the name "SOB"). > > Any scaling issues AOL is having has to do with how it has since been > misused and/or operationalized. > > -- Dossy > > -- > Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ > Panoptic Computer Network | http://panoptic.com/ > "He realized the fastest way to change is to laugh at your own > folly -- then you can let go and quickly move on." (p. 70) > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > <[EMAIL PROTECTED]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] HTTP client choices in AOLServer (was: RE: [AOLSERVER] nsdci query)
On 2007.10.25, John Caruso <[EMAIL PROTECTED]> wrote: > On Tuesday 10:30 AM 10/23/2007, Dossy Shiobara wrote: > >As part of the nsopenssl module, there are (IMHO, broken) > >implementations of ns_httpsopen and friends. > > Could you expand on what you think is broken about them? At AOL, a team that was using them had run into some kind of hang issue where a blocking Tcl [read] was done on the Tcl channel (that was backed by an nsopenssl SSL conn) that would never wake up and/or would time out. -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] HTTP client choices in AOLServer (was: RE: [AOLSERVER] nsdci query)
On Tuesday 10:30 AM 10/23/2007, Dossy Shiobara wrote: As part of the nsopenssl module, there are (IMHO, broken) implementations of ns_httpsopen and friends. Could you expand on what you think is broken about them? - John -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
In general, I find the SOB model works well. The caching on the server and the client provide unmatched performance. The NCF provides a mechanism to flush the cache on updates. The Devil is always in the details. How you set up multiple SOB servers and then get the publishes to multiple SOB servers can certainly create scaling and topology management issues. Most sites to do not have the demands that AOL has - and a simple SOB solution would be nice addition to any content distribution system. However there are advantages to stateless systems. :-) (How's that for diplomatic) M On Oct 25, 2007, at 10:19 AM, Dossy Shiobara wrote: On 2007.10.25, Jay Rohr <[EMAIL PROTECTED]> wrote: In general, I would not recommend using sob. It was developed for very specific requirements at AOL and it does not scale well under load - volume or frequency. We are in the process of replacing it with something else. Okay, this is just plain misinformation. SOB works very well for what it was intended for, a low-write, high-read, distributed service for small objects (thus, the name "SOB"). Any scaling issues AOL is having has to do with how it has since been misused and/or operationalized. -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Dossy Shiobara wrote: > On 2007.10.25, Jay Rohr <[EMAIL PROTECTED]> wrote: > >> In general, I would not recommend using sob. It was developed for very >> specific requirements at AOL and it does not scale well under load - >> volume or frequency. We are in the process of replacing it with >> something else. >> > > Okay, this is just plain misinformation. SOB works very well for what > it was intended for, a low-write, high-read, distributed service for > small objects (thus, the name "SOB"). > > Any scaling issues AOL is having has to do with how it has since been > misused and/or operationalized. > > That was my point :-) > -- Dossy > > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. begin:vcard fn:Jay J. Rohr n:Rohr;Jay email;internet:[EMAIL PROTECTED] x-mozilla-html:TRUE version:2.1 end:vcard
Re: [AOLSERVER] nsdci query
On Thursday 25 October 2007 07:19, Dossy Shiobara wrote: > On 2007.10.25, Jay Rohr <[EMAIL PROTECTED]> wrote: > > In general, I would not recommend using sob. It was developed for very > > specific requirements at AOL and it does not scale well under load - > > volume or frequency. We are in the process of replacing it with > > something else. > > Okay, this is just plain misinformation. SOB works very well for what > it was intended for, a low-write, high-read, distributed service for > small objects (thus, the name "SOB"). I would agree with this. Although I haven't used SOB, I have looked at the code. If anyone can improve upon reading static information from disk or memory, then they can outperform SOB. My understanding is that SOB simply 'compiles' dynamic information into a semi-static form. For instance, how often does the weather change in Seattle? Although the information is dynamic over large timescales, over the course of one day, not much changes. If you have 100 queries a second for Seattle weather, maybe put it into a SOB and let a backend service update it when needed. Just because you have dynamic data doesn't mean you have to shoot yourself in the foot when your service becomes popular. I wrote a small subscribe/push service that for some reason I called a tclbean. (I gave up waiting for NVs to be released by AOL). Essentially a server (as client) connects to the publisher and gives a callback address and a series of nsv arrays to subscribe to. The publish server connects back to the client. A scheduled proc on the publish server pushes updates. The original client can then always grab information from the nsv array and not worry about anything else. It is hard to imagine scalability problems with design, and I doubt there are any with SOB. But it would be interesting to hear what issues were found. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
On 2007.10.25, Jay Rohr <[EMAIL PROTECTED]> wrote: > In general, I would not recommend using sob. It was developed for very > specific requirements at AOL and it does not scale well under load - > volume or frequency. We are in the process of replacing it with > something else. Okay, this is just plain misinformation. SOB works very well for what it was intended for, a low-write, high-read, distributed service for small objects (thus, the name "SOB"). Any scaling issues AOL is having has to do with how it has since been misused and/or operationalized. -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
In general, I would not recommend using sob. It was developed for very specific requirements at AOL and it does not scale well under load - volume or frequency. We are in the process of replacing it with something else. Jay Damien O'Rourke wrote: > So ns_http handles binary but sob doesn't ... but sob could if it used > similar binary handling code to ns_http? > > Damien > > > On 23/10/2007, at 12:13 PM, Michael Andrews wrote: > >> That's what I believe as well. While ns_http is a different code base >> - I think both suffer from the same binary termination issue. I'm not >> sure if we saw this when using nsob with the old DCIRPC protocol - >> but we never used it for binary, so when we moved to HTTP we probably >> never thought it to be problematic. >> >> Check out dcicommon/rpc.c and dcicommon/server.c (I think). >> >> M >> >> On Oct 22, 2007, at 9:08 PM, Jeff Rogers wrote: >> >>> Dossy Shiobara wrote: > What I'm struggling with though is if I store an image file to the > sob with sob.copy and then try and retrieve it later with nsob.get, > it only returns the first 4 or 10 bytes of the file. SOB isn't encoding-aware and is actually really naive--it doesn't handle binary data well. >>> >>> Not sure if this it related, but I was just looking at ns_http for >>> an unrelated purpose and it appears that it is not binary clean at >>> all, since it uses Tcl_SetVar which expects a null-terminated >>> string. Getting binary files typically resulted in a 4 byte >>> response, could be that an average jpg file has a null at byte 4. >>> >>> -J >>> >>> >>> -- >>> AOLserver - http://www.aolserver.com/ >>> >>> To Remove yourself from this list, simply send an email to >>> <[EMAIL PROTECTED]> with the >>> body of "SIGNOFF AOLSERVER" in the email message. You can leave the >>> Subject: field of your email blank. >> >> >> -- >> AOLserver - http://www.aolserver.com/ >> >> To Remove yourself from this list, simply send an email to >> <[EMAIL PROTECTED]> with the >> body of "SIGNOFF AOLSERVER" in the email message. You can leave the >> Subject: field of your email blank. > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > <[EMAIL PROTECTED]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. begin:vcard fn:Jay J. Rohr n:Rohr;Jay email;internet:[EMAIL PROTECTED] x-mozilla-html:TRUE version:2.1 end:vcard
Re: [AOLSERVER] nsdci query
So ns_http handles binary but sob doesn't ... but sob could if it used similar binary handling code to ns_http? Damien On 23/10/2007, at 12:13 PM, Michael Andrews wrote: That's what I believe as well. While ns_http is a different code base - I think both suffer from the same binary termination issue. I'm not sure if we saw this when using nsob with the old DCIRPC protocol - but we never used it for binary, so when we moved to HTTP we probably never thought it to be problematic. Check out dcicommon/rpc.c and dcicommon/server.c (I think). M On Oct 22, 2007, at 9:08 PM, Jeff Rogers wrote: Dossy Shiobara wrote: What I'm struggling with though is if I store an image file to the sob with sob.copy and then try and retrieve it later with nsob.get, it only returns the first 4 or 10 bytes of the file. SOB isn't encoding-aware and is actually really naive--it doesn't handle binary data well. Not sure if this it related, but I was just looking at ns_http for an unrelated purpose and it appears that it is not binary clean at all, since it uses Tcl_SetVar which expects a null-terminated string. Getting binary files typically resulted in a 4 byte response, could be that an average jpg file has a null at byte 4. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] HTTP client choices in AOLServer (was: RE: [AOLSERVER] nsdci query)
On Tuesday 23 October 2007 09:07, Rick Cobb wrote: > ns_http (http://panoptic.com/wiki/aolserver/Ns_http) > Supports asynchronous operation > Supports all methods > Only supports BASIC auth > No proxy support > No SSL (https) support > Implemented largely in C > Available only in AOLServer 4 and above > Built-in > Only usable from a single thread (no channel/socket passing) This isn't quite accurate. What is going on in ns_http, is that a single thread does socket operations. When you do a ns_http queue, it places the request in a queue in this thread (different than a connection thread). The [ns_http queue] returns an id used during [ns_http wait]. [ns_http wait] can be called in a separate thread, like a scheduled proc. So you don't have to pass a socket around, it always remains in the queue thread. Also, you can add headers in a similar way to a regular request. That is, you provide a ns_set with additional headers. So whatever auth methods are supported with headers, you can do with ns_http. There is also ns_httpsget/post which handles https, just like ns_httpget/post. The commands ns_httpopen/ns_httpsopen are use by the get/post indirectly. I also think these may follow redirects. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] HTTP client choices in AOLServer (was: RE: [AOLSERVER] nsdci query)
On 2007.10.23, Rick Cobb <[EMAIL PROTECTED]> wrote: > So, for example, if I want to communicate with a RESTful application > (i.e., uses PUT, DELETE, etc) over HTTPS, it looks like there is no > option. Of course, most such applications also provide a POST wrapper, > but didn't the way-back Navi/AOLServer team have a solution for this? As part of the nsopenssl module, there are (IMHO, broken) implementations of ns_httpsopen and friends. I think TclCurl will do SSL if the underlying libcurl is compiled with support for it, too. -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Michael Andrews wrote: I think the ns_http bug was fixed in the HEAD by Nate. *ashamed* Yeah, the code I'm looking at is 4.0.8, pretty old. I probably just need to upgrade and this will work better. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] HTTP client choices in AOLServer (was: RE: [AOLSERVER] nsdci query)
It seems to me like in AOLServer 4, we have four ways for Tcl code to
behave as an HTTP client. Has anybody done any compare & contrast on
them? As far as I can tell without a lot of code experience on some of
them, we have:
ns_http{open,get,post}
Generally synchronous
Only supports GET and POST
Only supports BASIC auth
No proxy support
No SSL (https) support
Implemented entirely in Tcl
Best compatibility (available in AOLServer 2 - 4)
Built-in
Only usable from a single thread (no channel/socket passing)
ns_http (http://panoptic.com/wiki/aolserver/Ns_http)
Supports asynchronous operation
Supports all methods
Only supports BASIC auth
No proxy support
No SSL (https) support
Implemented largely in C
Available only in AOLServer 4 and above
Built-in
Only usable from a single thread (no channel/socket passing)
Tcl's HTTP package (the one I know the least about)
Supports asynchronous operation with progress callbacks
Supports GET, POST, and HEAD
Not obvious how to use any authentication other than BASIC
Supports proxy on a per-call basis (but you're responsible for .pac
translation if necessary).
Supports TLS/SSL (via callback)
Somewhat more complicated API to use than the above options
Built-in since Tcl 7?
TclCurl
(http://personal1.iddeo.es/andresgarci/tclcurl/english/docs.html)
Supports asynchronous operation
Supports GET and POST, not clear about others but don't look
supported
Supports BASIC, DIGEST, GSS-Negotiate, NTLM
Supports proxy on a per-call basis (but you're responsible for .pac
translation if necessary)
Implemented in C
Requires packaging work (bringing it into your Tcl & AOLServer
build)
Multi-threading requires config / build - time work (c-ares)
Passing handle between threads requires wrapping in your own mutex
Supports FTP, telnet, other common URLs
Substantially more complicated API to use than the other options
So, for example, if I want to communicate with a RESTful application
(i.e., uses PUT, DELETE, etc) over HTTPS, it looks like there is no
option. Of course, most such applications also provide a POST wrapper,
but didn't the way-back Navi/AOLServer team have a solution for this?
Thanks --
-- ReC
-Original Message-
From: AOLserver Discussion [mailto:[EMAIL PROTECTED] On Behalf
Of Michael Andrews
Sent: Tuesday, October 23, 2007 7:44 AM
To: AOLSERVER@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] nsdci query
I think the ns_http bug was fixed in the HEAD by Nate.
On Oct 23, 2007, at 9:31 AM, Tom Jackson wrote:
> Jeff,
>
> I have ns_http working, at least for me here with binary data.
> Supposedly this
> is the reason for ns_http (not ns_httpget/post or ns_httpsget/post):
>
> Here is a script:
>
> set id [ns_http queue http://192.168.1.102:8000/sns-thumb.jpg]
>
> set ok [ns_http wait -result image -status status $id]
>
> if {"$status" == "200"} {
> set fd [open [file join [file dirname [info script]]
> myimage.jpg] w+]
> } else {
> ns_return $status text/html $image
> return -code return
> }
>
> # Very important:
> fconfigure $fd -translation binary
>
> puts $fd $image
>
> close $fd
>
> ns_return $status text/html "Status: $status
> My Image"
>
>
>
>
> On Monday 22 October 2007 18:08, Jeff Rogers wrote:
>> Dossy Shiobara wrote:
>>>> What I'm struggling with though is if I store an image file to the
>>>> sob with sob.copy and then try and retrieve it later with nsob.get,
>>>> it only returns the first 4 or 10 bytes of the file.
>>>
>>> SOB isn't encoding-aware and is actually really naive--it doesn't
>>> handle
>>> binary data well.
>>
>> Not sure if this it related, but I was just looking at ns_http for an
>> unrelated purpose and it appears that it is not binary clean at all,
>> since it uses Tcl_SetVar which expects a null-terminated string.
>> Getting binary files typically resulted in a 4 byte response,
>> could be
>> that an average jpg file has a null at byte 4.
>>
>> -J
>>
>>
>> --
>> AOLserver - http://www.aolserver.com/
>>
>> To Remove yourself from this list, simply send an email to
>> <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER"
>> in the
>> email message. You can leave the Subject: field of your email blank.
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to
> <[EMAIL PROTECTED]> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field
Re: [AOLSERVER] nsdci query
I think the ns_http bug was fixed in the HEAD by Nate.
On Oct 23, 2007, at 9:31 AM, Tom Jackson wrote:
Jeff,
I have ns_http working, at least for me here with binary data.
Supposedly this
is the reason for ns_http (not ns_httpget/post or ns_httpsget/post):
Here is a script:
set id [ns_http queue http://192.168.1.102:8000/sns-thumb.jpg]
set ok [ns_http wait -result image -status status $id]
if {"$status" == "200"} {
set fd [open [file join [file dirname [info script]]
myimage.jpg] w+]
} else {
ns_return $status text/html $image
return -code return
}
# Very important:
fconfigure $fd -translation binary
puts $fd $image
close $fd
ns_return $status text/html "Status: $status
My Image"
On Monday 22 October 2007 18:08, Jeff Rogers wrote:
Dossy Shiobara wrote:
What I'm struggling with though is if I store an image file to the
sob with sob.copy and then try and retrieve it later with nsob.get,
it only returns the first 4 or 10 bytes of the file.
SOB isn't encoding-aware and is actually really naive--it doesn't
handle
binary data well.
Not sure if this it related, but I was just looking at ns_http for an
unrelated purpose and it appears that it is not binary clean at all,
since it uses Tcl_SetVar which expects a null-terminated string.
Getting binary files typically resulted in a 4 byte response,
could be
that an average jpg file has a null at byte 4.
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
<[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER"
in the
email message. You can leave the Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
<[EMAIL PROTECTED]> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the
Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] nsdci query
Jeff,
I have ns_http working, at least for me here with binary data. Supposedly this
is the reason for ns_http (not ns_httpget/post or ns_httpsget/post):
Here is a script:
set id [ns_http queue http://192.168.1.102:8000/sns-thumb.jpg]
set ok [ns_http wait -result image -status status $id]
if {"$status" == "200"} {
set fd [open [file join [file dirname [info script]] myimage.jpg] w+]
} else {
ns_return $status text/html $image
return -code return
}
# Very important:
fconfigure $fd -translation binary
puts $fd $image
close $fd
ns_return $status text/html "Status: $status
My Image"
On Monday 22 October 2007 18:08, Jeff Rogers wrote:
> Dossy Shiobara wrote:
> >> What I'm struggling with though is if I store an image file to the
> >> sob with sob.copy and then try and retrieve it later with nsob.get,
> >> it only returns the first 4 or 10 bytes of the file.
> >
> > SOB isn't encoding-aware and is actually really naive--it doesn't handle
> > binary data well.
>
> Not sure if this it related, but I was just looking at ns_http for an
> unrelated purpose and it appears that it is not binary clean at all,
> since it uses Tcl_SetVar which expects a null-terminated string.
> Getting binary files typically resulted in a 4 byte response, could be
> that an average jpg file has a null at byte 4.
>
> -J
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to
> <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the
> email message. You can leave the Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] nsdci query
That's what I believe as well. While ns_http is a different code base - I think both suffer from the same binary termination issue. I'm not sure if we saw this when using nsob with the old DCIRPC protocol - but we never used it for binary, so when we moved to HTTP we probably never thought it to be problematic. Check out dcicommon/rpc.c and dcicommon/server.c (I think). M On Oct 22, 2007, at 9:08 PM, Jeff Rogers wrote: Dossy Shiobara wrote: What I'm struggling with though is if I store an image file to the sob with sob.copy and then try and retrieve it later with nsob.get, it only returns the first 4 or 10 bytes of the file. SOB isn't encoding-aware and is actually really naive--it doesn't handle binary data well. Not sure if this it related, but I was just looking at ns_http for an unrelated purpose and it appears that it is not binary clean at all, since it uses Tcl_SetVar which expects a null-terminated string. Getting binary files typically resulted in a 4 byte response, could be that an average jpg file has a null at byte 4. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Thanks Andrew. And what Jeff Rogers mentioned about ns_http not being binary friendly seems to tie in with that. The way I've got round it so far is to use the nsob.copy for loading up the files, and nproxy plus 2 level cache like the sob system for getting the image files. That way I can at least set the encoding when reading and returning. It would have been nicer if the sob system handled it all though ... Damien On 23/10/2007, at 10:47 AM, Michael Andrews wrote: nsob is mostly used for text content. While the "copy" would work with a binary file - a "get" would probably not work with a binary file since it is set up over HTTP and the binary code might terminate the HTTP response before intended. But that is just a guess. I have not really read the RPC/HTTP code in nsdci. M On Oct 22, 2007, at 7:28 PM, Damien O'Rourke wrote: Hi, I've been experimenting with the components in nsdci, most particularly the sob service, and am impressed with it. What I'm struggling with though is if I store an image file to the sob with nsob.copy and then try and retrieve it later with nsob.get, it only returns the first 4 or 10 bytes of the file. If I do a nsob.put I can then do a nsob.get successfully, but the image file stored by the nsob.put command is not viewable with other image viewers. I suspect this is an encoding/binary issue but haven't been able to figure out the where and whyfor. Any ideas on where the issue might be or whether it's even possible to do what I want using the sob component? Thanks Damien O'Rourke -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Dossy Shiobara wrote: What I'm struggling with though is if I store an image file to the sob with sob.copy and then try and retrieve it later with nsob.get, it only returns the first 4 or 10 bytes of the file. SOB isn't encoding-aware and is actually really naive--it doesn't handle binary data well. Not sure if this it related, but I was just looking at ns_http for an unrelated purpose and it appears that it is not binary clean at all, since it uses Tcl_SetVar which expects a null-terminated string. Getting binary files typically resulted in a 4 byte response, could be that an average jpg file has a null at byte 4. -J -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
Dossy, thanks for the quick response. I didn't find any issues in getting it working ... even splitting the components across different servers. And possibly I didn't explain my issue adequately as what I can do successfully at the moment is - have an image uploaded via web page to frontend server (which saves it as temp file) - open and load the temp file as binary into a variable and send that to the sob server with nsob.put -- This stores it ... but not in binary format (i believe) as I can't view that image with some other image viewer, but I can view it in web page using nsob.get. So I think it's already in utf-8 format, which is why it works. But what doesn't work is - open and load the temp file as binary into a variable and send that to the sob server with nsob.copy -- This stores it ... and in the original binary format as it's viewable with other image viewers. But is not viewable using nsob.get I really need to keep the binary format when it's stored by the sob server ... so the images can be viewed by other apps ... and also be viewable using nsob.get. Where it seems to be failing at the moment is, even though the sob server stored the data as binary (using nsob.copy), it doesn't seem to be able to read it in and return it using nsob.get. So I think nsob.get needs to be able to convert from binary to utf-8 to successfully return it ... ?? Thanks Damien O'Rourke [EMAIL PROTECTED] On 23/10/2007, at 10:11 AM, Dossy Shiobara wrote: On 2007.10.23, Damien O'Rourke <[EMAIL PROTECTED]> wrote: I've been experimenting with the components in nsdci, most particularly the sob service, and am impressed with it. I'm SO glad you were able to get it working! I keep hearing "we need documentation first" from people, but apparently there was enough for you to get it going, at least. What I'm struggling with though is if I store an image file to the sob with sob.copy and then try and retrieve it later with nsob.get, it only returns the first 4 or 10 bytes of the file. SOB isn't encoding-aware and is actually really naive--it doesn't handle binary data well. A work-around that may work is to use Tcl's [encoding] explicitly, i.e.: nsob.put $id [encoding convertto utf-8 $value] I believe if you do this, you shouldn't need to do anything special when you nsob.get, as Tcl internally expects the data to be utf-8. -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
nsob is mostly used for text content. While the "copy" would work with a binary file - a "get" would probably not work with a binary file since it is set up over HTTP and the binary code might terminate the HTTP response before intended. But that is just a guess. I have not really read the RPC/HTTP code in nsdci. M On Oct 22, 2007, at 7:28 PM, Damien O'Rourke wrote: Hi, I've been experimenting with the components in nsdci, most particularly the sob service, and am impressed with it. What I'm struggling with though is if I store an image file to the sob with nsob.copy and then try and retrieve it later with nsob.get, it only returns the first 4 or 10 bytes of the file. If I do a nsob.put I can then do a nsob.get successfully, but the image file stored by the nsob.put command is not viewable with other image viewers. I suspect this is an encoding/binary issue but haven't been able to figure out the where and whyfor. Any ideas on where the issue might be or whether it's even possible to do what I want using the sob component? Thanks Damien O'Rourke -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsdci query
On 2007.10.23, Damien O'Rourke <[EMAIL PROTECTED]> wrote: > I've been experimenting with the components in nsdci, most > particularly the sob service, and am impressed with it. I'm SO glad you were able to get it working! I keep hearing "we need documentation first" from people, but apparently there was enough for you to get it going, at least. > What I'm struggling with though is if I store an image file to the > sob with sob.copy and then try and retrieve it later with nsob.get, > it only returns the first 4 or 10 bytes of the file. SOB isn't encoding-aware and is actually really naive--it doesn't handle binary data well. A work-around that may work is to use Tcl's [encoding] explicitly, i.e.: nsob.put $id [encoding convertto utf-8 $value] I believe if you do this, you shouldn't need to do anything special when you nsob.get, as Tcl internally expects the data to be utf-8. -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
