Re: [AOLSERVER] Charset differences between 3.3+ad13 and 4.0.10?
Hi Janine,
My name is Maria and I am Tom wife, he need your your help, Can you
help him? call me at 425-241-8218 (cell) or 425-614-3105 (home)
Please call. My email is maria.aghda...@gmail.com. I need your your
help to updated some website, please call,
thanks
maria
On Thu, Dec 2, 2010 at 10:29 PM, Janine Ohmer jan...@furfly.net wrote:
Changing the charset seems to have fixed it, after a spot check of some
pages I know had problems. If the client doesn't find more, we'll be in
good shape.
Thanks, everyone!
janine
On Dec 2, 2010, at 2:08 AM, Bernhard van Woerden wrote:
Just found the same as Russell, it started life as a single byte but needs
some conversion to match it with the correct unicode character which can
then be stored in utf-8.
What happens if you ask AOLserver to output iso-8859-1 does it convert 0xc2
0x92 to 0x92 ?
ns_param OutputCharset iso-8859-1
On 2 December 2010 07:39, Janine Ohmer jan...@furfly.net wrote:
On Dec 1, 2010, at 11:16 AM, Bernhard van Woerden wrote:
Can you check the byte sequence of the string in the db that's causing a
problem.
Sort of... the column in question is of type text, and the only function I
can find that will convert from text to bytea is decode, not encode. So
here's what I did:
select decode(answer_3, 'escape') from public_places where
public_place_id=1012;
The interesting part of the result is Park\302\222s, which is supposed
to be Park's.
I then ran
select encode('\\302\\222':bytea, 'hex');
and got c292.
Google tells me that this is the same as U+0092, which is defined simply
as control, or private use 2.
This doesn't seem very helpful, but it's possible that my initial
assumptions on how to do this were wrong and I'm really just trying to look
up garbage data. :)
Bernhard, does this give you any clues?
thanks,
janine
select encode(column_name::bytea,'hex') from ...
or use get_byte to isolate
On 1 December 2010 18:44, Janine Ohmer jan...@furfly.net wrote:
Hi Brian,
The old system isn't accessible anymore (something went very wrong and
even though it's RAID and all, fsck ended up removing a bunch of files and
making the system unbootable). But from looking at the files in etc from
the backups I can deduce that LANG was set to the same thing as it is on the
new system, en_US.UTF-8.
I had tried setting these:
ns_param HackContentType 1
ns_param URLCharset utf-8
ns_param OutputCharset utf-8
ns_param HttpOpenCharset utf-8
Which did not work. I will try adding DefaultCharset just in case that
is the key, but I'm not holding my breath...
There is an environment variable I can set, PGCLIENTENCODING, which will
let me specify what encoding to use for the client, so I can add that to
nsd-postgres. But I'm not sure what to set it to - everything's already in
UTF8, so there shouldn't *be* any encoding issues (famous last words, I know
:).
I can't restart the site during the day (very picky client) so will not
be able to try anything until late afternoon.
thanks,
janine
On Dec 1, 2010, at 3:32 AM, Fenton, Brian wrote:
Hi Janine
this may not be the answer but it can't do any harm. You should
consider adding these to the AOLserver tcl file:
ns_param HackContentType 1
ns_param DefaultCharset utf-8
ns_param HttpOpenCharset utf-8
ns_param OutputCharset utf-8
ns_param URLCharset utf-8
Also, rule out any OS differences by checking your locale, LANG etc.
Does Postgres have an equivalent to Oracle's NLS_LANG? Hopefully you won't
have to dive into codepages and all that stuff!
best wishes
Brian Fenton
From: AOLserver Discussion [AOLSERVER@LISTSERV.AOL.COM] On Behalf Of
Janine Ohmer [jan...@furfly.net]
Sent: 01 December 2010 05:19
To: AOLSERVER@LISTSERV.AOL.COM
Subject: [AOLSERVER] Charset differences between 3.3+ad13 and 4.0.10?
Me again... still working out the last details on those sites I had to
move.
We're having some issues with characters like apostrophes and dashes
either disappearing (Safari) or showing up as garbage characters
(Firefox).
I'm using the same version of Postgres and the same codebase. The
databases on both systems use the UNICODE encoding. The main thing that's
different is the AOLserver version. There isn't anything about charset in
either the old or the new config file.
Has anyone done this conversion (in recent memory, that is :) and knows
what the problem might be?
thanks,
janine
---
Janine Ohmer (formerly Sisk)
President/CEO of furfly, LLC
503-693-6407
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the
Subject: field of your email blank.
--
AOLserver -
Re: [AOLSERVER] Does not work ns_return + zlib
True, but since gzip is something which can be applied at any point
between the source and destination, you have to wonder why the mobile
service provider isn't doing this for their customers.
If your website/service actually serves up content for mobile
customers, it would also help to generate scaled down content for
these users. This is a much bigger change than simply compressing the
content, but many web pages seem unusable in on small device screens,
heck some of the triple column news sites are almost impossible on a
laptop.
Maybe we should look at putting this into ns_return/fastpath if Brett
has a list of the changes.
tom jackson
Brett: you could email me the changed source files instead of
providing a patch to some git or cvs version.
On Sun, Dec 19, 2010 at 2:39 AM, Mark Aufflick
mark-aolser...@aufflick.com wrote:
Hi Tom,
Notwithstanding your legitimate issue that gzipping every html and css
file on the fly is counterproductive in many cases, one case this is
not true is serving to mobile devices - if you're on the end of a weak
GPRS connection with a fairly powerful cpu, you are going to notice
the difference between gzipped and ungzipped content.
Just my 2c :)
--
Mark Aufflick
http://mark.aufflick.com/about/contact
http://pumptheory.com/about
On Tue, Dec 7, 2010 at 9:51 AM, Tom Jackson t...@rmadilo.com wrote:
On Mon, Dec 6, 2010 at 9:13 AM, Alexey Pechnikov pechni...@mobigroup.ru
wrote:
I'm see the code in connio.c:
/*
* GZIP the content when not streaming if enabled and the content
* length is above the minimum.
*/
if (!stream
(conn-flags NS_CONN_GZIP)
(servPtr-opts.flags SERV_GZIP)
(len (int) servPtr-opts.gzipmin)
(ahdr = Ns_SetIGet(conn-headers, Accept-Encoding)) != NULL
strstr(ahdr, gzip) != NULL
Ns_Gzip(buf, len, servPtr-opts.gziplevel, gzip) == NS_OK) {
buf = gzip.string;
len = gzip.length;
Ns_ConnCondSetHeaders(conn, Content-Encoding, gzip);
}
There are no checks for content-type and older version of Internet Explorer
(IE5, IE6 and may be IE7 have a lot of problems with gzipped scripts and
styles).
And yet your examples provided even less customization. There is
almost no reason to waste cpu on compressing output, just provide a
gzipped file for very large files. Who are you trying to save money
for anyway?
I don't think that this code is useful for production.
Right, then don't use it.
And we may
add ETAG functionality and smart caching checksums of results for decreasing
data transfer and server loading. I dont know about your situation but my
clients have limited internet connections (especially on mobile devices) and
ETAG header transmitting is faster when gzipped content...
Then the least of your problems is gzipping content, you need to
actively minimize the data transfered. But all of this sounds like
_your_ problem, not the problem of a generic application server. You
haven't even figured out how automatic compression works in AOLserver
yet you want to propose additional features.
For static files
on group of hosts application-defined ETAG is helpful too but internal AOL
last-modified-since mechanizm is niot useful (it's not the AOL problem, of
cource).
??
P.S. I dont understand why my suggestion to complete AOL documentation by
examples was produce the holywar about Tcl 8.4 vs 8.5 vs 8.6.
Because I'm a jerk and overreact to what I think are idiotic statements?
BTW, you didn't provide any useful additions to AOLserver documentation.
I think AOL
4.5.1 + Tcl 8.5 is better choice for new projects and Tcl 8.6 is better for
some utilities (fast internal base64 realization, half-closed sockets and
other features help me to build faster applications with a few lines of
code).
I agree with Gustaf: the latest 8.5 is worth the effort. There are
certain features which simplify very annoying code. This is true even
if your version of 8.5 is slower than 8.4. But you have to actively
update your code to take advantage of the new features. The more code
you have, the less benefit you get from upgrading without code
conversion. However, Gustaf mentioned a higher stability in 8.5. This
could easily override the limited benefit of simply moving the Tcl
library to 8.5 from 8.4.
Tcl 8.6 documentation of zlib functions is much better than AOL
documentation of ns_zlib module and some of this docs and examples can be
helpful for AOL, why not?
If you use Tcl, use the Tcl documentation. If you use AOLserver, use
the AOLserver documentation. I'm not sure why you keep confusing these
two things.
tom jackson (AKA the jerk)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list
Re: [AOLSERVER] Does not work ns_return + zlib
On Mon, Dec 6, 2010 at 9:13 AM, Alexey Pechnikov pechni...@mobigroup.ru wrote:
I'm see the code in connio.c:
/*
* GZIP the content when not streaming if enabled and the content
* length is above the minimum.
*/
if (!stream
(conn-flags NS_CONN_GZIP)
(servPtr-opts.flags SERV_GZIP)
(len (int) servPtr-opts.gzipmin)
(ahdr = Ns_SetIGet(conn-headers, Accept-Encoding)) != NULL
strstr(ahdr, gzip) != NULL
Ns_Gzip(buf, len, servPtr-opts.gziplevel, gzip) == NS_OK) {
buf = gzip.string;
len = gzip.length;
Ns_ConnCondSetHeaders(conn, Content-Encoding, gzip);
}
There are no checks for content-type and older version of Internet Explorer
(IE5, IE6 and may be IE7 have a lot of problems with gzipped scripts and
styles).
And yet your examples provided even less customization. There is
almost no reason to waste cpu on compressing output, just provide a
gzipped file for very large files. Who are you trying to save money
for anyway?
I don't think that this code is useful for production.
Right, then don't use it.
And we may
add ETAG functionality and smart caching checksums of results for decreasing
data transfer and server loading. I dont know about your situation but my
clients have limited internet connections (especially on mobile devices) and
ETAG header transmitting is faster when gzipped content...
Then the least of your problems is gzipping content, you need to
actively minimize the data transfered. But all of this sounds like
_your_ problem, not the problem of a generic application server. You
haven't even figured out how automatic compression works in AOLserver
yet you want to propose additional features.
For static files
on group of hosts application-defined ETAG is helpful too but internal AOL
last-modified-since mechanizm is niot useful (it's not the AOL problem, of
cource).
??
P.S. I dont understand why my suggestion to complete AOL documentation by
examples was produce the holywar about Tcl 8.4 vs 8.5 vs 8.6.
Because I'm a jerk and overreact to what I think are idiotic statements?
BTW, you didn't provide any useful additions to AOLserver documentation.
I think AOL
4.5.1 + Tcl 8.5 is better choice for new projects and Tcl 8.6 is better for
some utilities (fast internal base64 realization, half-closed sockets and
other features help me to build faster applications with a few lines of
code).
I agree with Gustaf: the latest 8.5 is worth the effort. There are
certain features which simplify very annoying code. This is true even
if your version of 8.5 is slower than 8.4. But you have to actively
update your code to take advantage of the new features. The more code
you have, the less benefit you get from upgrading without code
conversion. However, Gustaf mentioned a higher stability in 8.5. This
could easily override the limited benefit of simply moving the Tcl
library to 8.5 from 8.4.
Tcl 8.6 documentation of zlib functions is much better than AOL
documentation of ns_zlib module and some of this docs and examples can be
helpful for AOL, why not?
If you use Tcl, use the Tcl documentation. If you use AOLserver, use
the AOLserver documentation. I'm not sure why you keep confusing these
two things.
tom jackson (AKA the jerk)
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Does not work ns_return + zlib
Here's a temp link to a git version of the module: http://junom.com/gitweb/gitweb.perl?p=aolserver.git;a=tree;f=nsreturnz;h=49d6471737b294;hb=master (use the snapshot link on this page to get a zip or tgz of just this module (unfortunately the enclosing directory will be misnamed). I've already done a cvsimport on all the modules. I'll put them up on github later today. tom jackson On Wed, Dec 1, 2010 at 5:06 AM, Alexey Pechnikov pechni...@mobigroup.ru wrote: Can you help me to resolv the problem? ns_returnz: compress2 failed, status=-5 -- Best regards, Alexey Pechnikov. http://pechnikov.tel/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Does not work ns_return + zlib
BTW, I think the functionality in the external module nsreturnz is now part of nszlib, which should be in your git repo, otherwise here's a link: http://junom.com/gitweb/gitweb.perl?p=aolserver.git;a=tree;f=nszlib;h=0ac26233;hb=origin tom jackson On Wed, Dec 1, 2010 at 7:45 AM, Tom Jackson t...@rmadilo.com wrote: Here's a temp link to a git version of the module: http://junom.com/gitweb/gitweb.perl?p=aolserver.git;a=tree;f=nsreturnz;h=49d6471737b294;hb=master (use the snapshot link on this page to get a zip or tgz of just this module (unfortunately the enclosing directory will be misnamed). I've already done a cvsimport on all the modules. I'll put them up on github later today. tom jackson On Wed, Dec 1, 2010 at 5:06 AM, Alexey Pechnikov pechni...@mobigroup.ru wrote: Can you help me to resolv the problem? ns_returnz: compress2 failed, status=-5 -- Best regards, Alexey Pechnikov. http://pechnikov.tel/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Does not work ns_return + zlib
I don't use this feature, it just puts extra work on the server and
client to save something. If service providers think it is necessary
to save bandwidth, they could probably compress the content
themselves. But from looking at the code, you shouldn't have to do
anything except configure the module to get transparent compression.
You have to set minimum file content size, etc. and adp needs
independent configuration if you use adps. In other words, you don't
have to edit your site code to get compression with ns_zlib.
tom jackson
On Wed, Dec 1, 2010 at 9:11 AM, Alexey Pechnikov pechni...@mobigroup.ru wrote:
2010/12/1 Tom Jackson t...@rmadilo.com
BTW, I think the functionality in the external module nsreturnz is now
part of nszlib
But are you sure that this works? In my tests only ns_returnfile work
correct for gzipped content:
ns_register_proc GET /test ad_test_proc
proc ad_test_proc {ignore} {
ns_updateheader Content-Encoding gzip
ns_return 200 text/plain [ns_zlib gzip test]
}
$ curl http://localhost:8200/test 2/dev/null| hexdump
000 c21f 088b 2b03 2d49 012e
010 0c00 7f7e 98c3 0004
01a
ns_register_proc GET /test ad_test_proc
proc ad_test_proc {ignore} {
set fd [open /tmp/test.gz r]
fconfigure $fd -translation binary -encoding binary
ns_updateheader Content-Encoding gzip
ns_return 200 text/plain [read $fd]
close $fd
}
$ echo -n test|gzip/tmp/test.gz
$ curl http://localhost:8200/test 2/dev/null| hexdump
000 c21f 088b c300 7f99 b6c3 004c 2b03 2d49
010 012e 0c00 7f7e 98c3 0004
01c
ns_register_proc GET /test ad_test_proc
proc ad_test_proc {ignore} {
ns_updateheader Content-Encoding gzip
ns_returnfile 200 text/plain /tmp/test.gz
}
$ echo -n test|gzip/tmp/test.gz
$ curl http://localhost:8200/test 2/dev/null| hexdump
000 8b1f 0008 7fd9 4cf6 0300 492b 2e2d 0001
010 7e0c d87f 0004
018
Browsers can show only last question result. I don't understand how ns_zlib
encode the data.
--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Does not work ns_return + zlib
I must be missing something. What does the below code have to do with
AOLserver or ns_zlib?
Personally I also wouldn't assume that AOLserver works perfectly with
Tcl8.6. Unless you absolutely need 8.5, I would stick with the latest
8.4 version. For one thing 8.4 is faster than either.
tom jackson
2010/12/1 Alexey Pechnikov pechni...@mobigroup.ru:
But does not work for cyrillic strings. And zlib in Tcl 8.6 produce wrong
result too:
set content привет
set gzip [binary format H*iH* 1f8b0800 [clock seconds] 0003]
append gzip [zlib deflate $content 0]
append gzip [binary format i [zlib crc32 $content]]
append gzip [binary format i [string length $content]]
set fd [open /tmp/test.gz w]
fconfigure $fd -translation binary -encoding binary
puts -nonewline $fd $gzip
close $fd
puts [exec zcat /tmp/test.gz ]
�...@825b
But for latin symbols all right:
set content hello
set gzip [binary format H*iH* 1f8b0800 [clock seconds] 0003]
append gzip [zlib deflate $content]
append gzip [binary format i [zlib crc32 $content]]
append gzip [binary format i [string length $content]]
set fd [open /tmp/test.gz w]
fconfigure $fd -translation binary -encoding binary
puts -nonewline $fd $gzip
close $fd
puts [exec zcat /tmp/test.gz ]
hello
2010/12/1 Alexey Pechnikov pechni...@mobigroup.ru
This work correct:
ns_register_proc GET /test ad_test_proc
proc ad_test_proc {ignore} {
set gzip [ns_zlib gzip test]
set time [ns_httptime [ns_time]]
ns_write HTTP/1.0 200 OK
Content-Type: text/plain; charset=utf-8
Content-Encoding: gzip
\n
ns_write $gzip
}
$ curl http://localhost:8200/test 2/dev/null| hexdump
000 8b1f 0008 0300 492b 2e2d 0001
010 7e0c d87f 0004
018
But is possible to disable all output translations by ns_return? Where is
documented how ns_return process the data?..
--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Does not work ns_return + zlib
All you are doing here is using ns_zlib in Tcl. Who cares if this
works or not? It is possible that HTTP content compression is slightly
different than what is produced by ns_zlib. But this doesn't matter
really. What you should be testing is ns_returnz (sp?) or ns_return
(or adp output) with transparent compression (which takes into account
client capabilities, you can't just blindly send compressed content).
I remember looking into ns_zlib just because it was part of the adp
processing/fastpath code. I don't have any reason to believe that, if
configured correctly it doesn't work. I also doubt that ns_zlib would
have wound up in the core unless it worked, the author is well known
and I've never heard of anyone suggesting any of the repo modules
don't work.
I'll try to look into how ns_zlib is supposed to work. Maybe this is a
total lack of documentation problem.
tom jackson
On Wed, Dec 1, 2010 at 10:19 AM, Alexey Pechnikov
pechni...@mobigroup.ru wrote:
This work correct:
ns_register_proc GET /test ad_test_proc
proc ad_test_proc {ignore} {
set gzip [ns_zlib gzip test]
set time [ns_httptime [ns_time]]
ns_write HTTP/1.0 200 OK
Content-Type: text/plain; charset=utf-8
Content-Encoding: gzip
\n
ns_write $gzip
}
$ curl http://localhost:8200/test 2/dev/null| hexdump
000 8b1f 0008 0300 492b 2e2d 0001
010 7e0c d87f 0004
018
But is possible to disable all output translations by ns_return? Where is
documented how ns_return process the data?..
--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Does not work ns_return + zlib
2010/12/1 Jeff Hobbs je...@activestate.com: On 01/12/2010 12:31 PM, Tom Jackson wrote: Personally I also wouldn't assume that AOLserver works perfectly with Tcl8.6. Unless you absolutely need 8.5, I would stick with the latest 8.4 version. For one thing 8.4 is faster than either. That's no longer true. Not sure what is no longer true. AOLserver is pretty much ahead of Tcl wrt client/server programming (and about 15 year ahead of Java). Tcl 8.6 continues to head into the direction of poor performance and useless new API. In case programmers haven't figured it out, multi-core processors are the norm, so stuff like thread bound coroutines are useless wrt performance and much harder to program than C thread based servers. But if you are stuck in one thread, maybe...maybe there is some benefit in moving to Tcl 8.6, unless you are using AOLserver. Please don't listen to the Tcl gurus, they are behind the curve...actually they are actively slowing down the curve. The evidence is simple: 8.4 is faster than 8.5 which is faster than 8.6. How is this even possible? The answer is simple: the developers do no know what they are doing. The evidence is the performance. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] Possible bug in ns_internalredirect, plus behavior difference from ns_rewriteurl
This is just a quick note to anyone using ns_internalredirect, especially as a replacement for ns_rewriteurl, provided by the C module nsrewrite. Background: ns_rewriteurl is a simple Tcl command wrapper around Ns_SetRequestUrl(). The result of the rewrite is that all following filter matching and/or registered proc matching are based upon the new url. ns_internalredirect does something different. It wraps Ns_ConnRedirect(). This function keeps track of recursion loops, calls Ns_SetRequestUrl() and then actually performs a mini-request. The mini-request does two things: runs the authorization proc for the new url and if OK, runs the request. So what are the actual and potential problems with this? My guess is that this works fine if Ns_ConnRedirect() is called in the registered proc state, but I'm not sure that even this is safe. What is an observable error/bug is when ns_internalredirect is called within a filter prior to choosing and running the registered proc. What I see in log files is two invocations of the registered proc, although only one is actually returned to the client. This behavior could lead to application misbehavior or even failure. For instance, you might POST data twice. However, this is an easily observed problem. A more serious issue is the potential short circuiting of certain code paths which were planned before a request runs. Imagine that a preauth filter calls ns_internalredirect. What happens is that this causes a mini-request which skips all remaining preauth filters and all postauth filters before running the registered proc. Then...I guess, this mini-request terminates and returns to the filter (usually with NS_OK). The filter then probably returns FILTER_OK and the original request, with a new url proceeds. Eventually the original request chooses the same registered proc and runs it. Then trace and cleanup filters run, covering up the damage. There are a large number of simple circumstances where this works okay, but they all require that the ns_internalredirect was called during the running of the registered proc. This is because when the mini-request returns, there is no more code to run, the registered proc has already been chosen so the state machine moves on to trace filters. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_httptime -1 crashes AOLserver on Windows
I'll just note that Ns_HttpTime, which is called by ns_httptime is
also called for nearly every request in Ns_ConnSetRequiredHeaders
Ns_ConnSetRequiredHeaders --
384 *
385 * Set a sane set of minimal headers for any response:
386 * MIME-Version, Date, Server, Content-Type, Content-Length
410 Ns_ConnCondSetHeaders(conn, Date, Ns_HttpTime(ds, NULL));
So if your server is only crashing in ns_httptime, that should narrow
down the source of the bug.
tom jackson
On Tue, Nov 23, 2010 at 2:21 AM, Fenton, Brian brian.fen...@quest.ie wrote:
Nice workaround Tom.
It actually kills the AOLserver process instantly, requiring a restart.
Brian
From: AOLserver Discussion [aolser...@listserv.aol.com] On Behalf Of Tom
Jackson [...@rmadilo.com]
Sent: 22 November 2010 19:02
To: AOLSERVER@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] ns_httptime -1 crashes AOLserver on Windows
Quick workaround for your problem:
proc ::ns_time { } {
return [clock seconds]
}
proc ::ns_httptime { seconds } {
return [clock format $seconds -format %a, %d %b %Y %H:%M:%S %Z -gmt 1]
}
Just redefine ns_httptime.
tom jackson
(However, does the C code crash, or just the tcl API?)
On Mon, Nov 22, 2010 at 9:17 AM, Fenton, Brian brian.fen...@quest.ie wrote:
Hi
My Windows version of AOLserver crashes hard if I call ns_httptime with a
value of -1. The version is
AOLserver/4.5.0 (aolserver4_5)
CVS Tag: $Name: aolserver_v45_r0 $
Built: May 2 2008 at 14:54:44
Tcl version: 8.5
Platform: win32
Can anyone else confirm this problem? I realise there is a more recent
Windows version of AOLserver - does anyone know if it is resolved there?
I'm getting it on Windows Server 2003 [Version 5.2.3790]
It seems to work fine on a unix version of AOLserver 4.5 that I tested.
thanks
Brian
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_httptime -1 crashes AOLserver on Windows
Quick workaround for your problem:
proc ::ns_time { } {
return [clock seconds]
}
proc ::ns_httptime { seconds } {
return [clock format $seconds -format %a, %d %b %Y %H:%M:%S %Z -gmt 1]
}
Just redefine ns_httptime.
tom jackson
(However, does the C code crash, or just the tcl API?)
On Mon, Nov 22, 2010 at 9:17 AM, Fenton, Brian brian.fen...@quest.ie wrote:
Hi
My Windows version of AOLserver crashes hard if I call ns_httptime with a
value of -1. The version is
AOLserver/4.5.0 (aolserver4_5)
CVS Tag: $Name: aolserver_v45_r0 $
Built: May 2 2008 at 14:54:44
Tcl version: 8.5
Platform: win32
Can anyone else confirm this problem? I realise there is a more recent
Windows version of AOLserver - does anyone know if it is resolved there?
I'm getting it on Windows Server 2003 [Version 5.2.3790]
It seems to work fine on a unix version of AOLserver 4.5 that I tested.
thanks
Brian
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] AOLserver 4.5.1 crashing with max connections per thread messages
Here is one example: See the full configuration at:
http://junom.com/document/aolserver/startup/errors.txt
ns_section ns/server/junom/pool/default
ns_param maxconnections 100
ns_param minthreads 4
ns_param maxthreads 10
ns_param threadtimeout 240
ns_param map {GET /}
ns_param map {POST /}
ns_section ns/server/junom/pool/fast
ns_param maxconnections 100
ns_param minthreads 2
ns_param maxthreads 10
ns_param threadtimeout 120
ns_param map {GET /*-thumb.jpg}
ns_section ns/server/junom/pools
ns_param default {default pool}
ns_param fast {fast pool}
There is also a pools.tcl file which goes in the global tcl library
directory (should be in the current AOLserver version).
Right now, this only works only to configure the default pool. I have
an updated pools.tcl file, but is also relies on a C patch which
creates a separate namespace for each virtual server. Otherwise, any
pool named default will overwrite the global pool, also named
default.
Here is what is in my pools.tcl file, which might give hints for using ns_pools:
set cfgsection ns/server/[ns_info server]
set minthreads [ns_config $cfgsection minthreads 0]
set maxthreads [ns_config $cfgsection maxthreads 10]
set maxconns [ns_config $cfgsection maxconnections 0]
set timeout [ns_config $cfgsection threadtimeout 0]
#ns_pools set default -minthreads $minthreads -maxthreads $maxthreads
-maxconns $maxconns -timeout $timeout
ns_log Notice Default Pool: [ns_pools get default]
# Setup optional threadpools
set poolSection $cfgsection/pools
set poolSet [ns_configsection $poolSection]
if {$poolSet ne } {
set poolSize [ns_set size $poolSet]
for {set i 0} {$i $poolSize} {incr i} {
set poolName [ns_set key $poolSet $i]
set poolDescription [ns_set value $poolSet $i]
set poolConfigSection ns/server/[ns_info server]/pool/$poolName
set poolConfigSet [ns_configsection $poolConfigSection]
if {$poolConfigSet eq } {
continue
}
set poolMinthreads [ns_config $poolConfigSection minthreads $minthreads]
set poolMaxthreads [ns_config $poolConfigSection maxthreads $maxthreads]
set poolMaxconns [ns_config $poolConfigSection maxconnections
$maxconns]
set poolTimeout[ns_config $poolConfigSection threadtimeout $timeout]
ns_pools set $poolName -minthreads $poolMinthreads -maxthreads
$poolMaxthreads -maxconns $poolMaxconns -timeout $poolTimeout
ns_log Notice $poolName Pool: [ns_pools get [ns_info
server]-$poolName]
set poolConfigSize [ns_set size $poolConfigSet]
for {set j 0} {$j $poolConfigSize} {incr j} {
if {[string tolower [ns_set key $poolConfigSet $j]] eq map} {
set mapList [split [ns_set value $poolConfigSet $j]]
set poolMethod [lindex $mapList 0]
set poolPattern [lindex $mapList 1]
ns_pools register ${poolName} [ns_info server] $poolMethod
$poolPattern
ns_log Notice ns_pools registered $poolName [ns_info server]
$poolMethod $poolPattern
}
}
}
}
(You can use the ns_pools command anywhere, even after startup or from
the control port)
I think the above script can be fixed by modifying [ns_pools set] to this:
ns_pools set [ns_info server]-$poolName -minthreads $poolMinthreads
-maxthreads $poolMaxthreads -maxconns $poolMaxconns -timeout
$poolTimeout
(anywhere you see $poolName, replace with [ns_info server]-$poolName):
ns_pools register [ns_info server]-${poolName} [ns_info server]
$poolMethod $poolPattern
tom jackson
(you can post or send me directly the config info during startup if
you have problems)
2010/11/19 Björn Þór Jónsson ban...@bthj.is:
Are there any good examples of proper connection threadpool configuration
available?
I've looked at http://openacs.org/forums/message-view?message_id=1146218 and
am a bit confused. (BTW this is not an OpenACS site, just plain .adp
pages).
Thanks Alexey for the Ubuntu packages, before checking them out I'll try to
get this latest AOLserver version just compiled to stay up :)
/Björn
2010/11/18 Tom Jackson t...@rmadilo.com
BTW, I don't think the issue you are seeing has anything to do with
the database pools, the problem is the connection threadpools (you are
using the default threadpool).
tom jackson
On Thu, Nov 18, 2010 at 8:42 AM, Alexey Pechnikov
pechni...@mobigroup.ru wrote:
On my debian lenny/squeeze 32 and 64 bit hosts this work fine with
PostgreSQL 8.1/8.4:
# Database drivers
ns_section ns/db/drivers
ns_param postgres nspostgres.so ;# An internal driver
#ns_section ns/db/driver/postgres
#ns_param pgbin /usr/lib/postgresql/8.1/bin/psql
ns_section ns/db/pools
ns_param maindb Main database Pool
ns_param session Session database Pool
ns_section ns/db/pool/maindb
ns_param driver
Re: [AOLSERVER] AOLserver 4.5.1 crashing with max connections per thread messages
My guess is that this is caused by the way the current threadpool code
works. I'll have to see if I can find my test data, but here is what I
remember off the top of my head.
There are a few causes:
1. use of Ns_CondSignal instead of Ns_CondBroadcast to wakeup
threadpool threads. This usually results in the thread which just sent
the signal to wake up and grab the mutex again, and service another
request. This results in the max number of requests per thread being
reached for a particular thread and it tries to exit.
2. The exiting thread starts up a replacement thread under certain
conditions. Sometimes, with many requests coming in, this new thread
will grab the mutex repeatedly, and get into the same condition as #1.
However, thread exiting from #1 hasn't yet exited and now its parent
is also trying to exit.
3. The basic problem is detecting when to allow threads to exit. For
instance, a thread might exit because it has been sitting around for
too long. Say it has serviced 10 connections and is supposed to exit
at 50. What has happened is that you remove the ability to handle 40
requests. The visible result of this is the inability of the server to
maintain threads between the min and max specified for a particular
threadpool. (Replacing a thread at thread exit patches over this
problem, but causes a different problem).
Note that it is hard to demonstrate the bug, I only found it by
hammering the server. But the bug inevitably shows up and crashes the
server. I also added additional logging code so I was able to track
what thread was servicing requests and the configuration of the thread
during each request (how many conns had been serviced, etc.).
tom jackson
2010/11/18 Björn Þór Jónsson ban...@bthj.is:
Hi,
After recently upgrading from AOLserver 4.5.0 to 4.5.1 and from
nspostgres-4.0 to nspostgres-4.1 the server is repeatedly crashing (when it
gets hammered by the google bots). The error.log has many entries like
these before the server dies:
[17/Nov/2010:02:18:42][700.3218660208][-default:6195-] Notice: exiting:
exceeded max connections per thread
[17/Nov/2010:02:18:43][700.3217636208][-default:6193-] Notice: exiting:
exceeded max connections per thread
[17/Nov/2010:02:18:44][700.3219172208][-default:6196-] Notice: exiting:
exceeded max connections per thread
[17/Nov/2010:02:18:45][700.3218148208][-default:6194-] Error: Tcl exception:
adp flush failed: connection closed
abort exception raised
while processing connection #31907:
GET ...
Host: localhost:8006
...
nsthreads: pthread_create failed in NsCreateThread: Resource temporarily
unavailable [this is the last line in the log before the crash]
This is the database section of the AOLserver config file:
ns_section ns/db/drivers
ns_param postgres nspostgres.so
ns_section ns/db/pools
ns_param pool1 Pool 1
ns_param pool2 Pool 2
ns_param pool3 Pool 3
ns_section ns/db/pool/pool1
ns_param maxidle 10
ns_param maxopen 10
ns_param connections 5
ns_param extendedtableinfo true
ns_param driver postgres
ns_param datasource localhost::${db_name}
ns_param user $user_account
ns_section ns/db/pool/pool2
ns_param maxidle 10
ns_param maxopen 10
ns_param connections 5
ns_param extendedtableinfo true
ns_param driver postgres
ns_param datasource localhost::${db_name}
ns_param user $user_account
ns_section ns/db/pool/pool3
ns_param maxidle 10
ns_param maxopen 10
ns_param connections 5
ns_param extendedtableinfo true
ns_param driver postgres
ns_param datasource localhost::${db_name}
ns_param user $user_account
ns_section ns/server/${server}/db
ns_param pools *
ns_param defaultpool pool1
The server is running on Ubuntu 10.04.1 LTS
2.6.32-25-generic-pae #45-Ubuntu SMP Sat Oct 16 21:01:33 UTC 2010 i686
GNU/Linux
Is there anything I should configure differently or has any other ideas what
might be causing this?
/Björn
--
Björn Þór Jónsson
http://bthj.is
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] AOLserver 4.5.1 crashing with max connections per thread messages
BTW, I don't think the issue you are seeing has anything to do with
the database pools, the problem is the connection threadpools (you are
using the default threadpool).
tom jackson
On Thu, Nov 18, 2010 at 8:42 AM, Alexey Pechnikov
pechni...@mobigroup.ru wrote:
On my debian lenny/squeeze 32 and 64 bit hosts this work fine with
PostgreSQL 8.1/8.4:
# Database drivers
ns_section ns/db/drivers
ns_param postgres nspostgres.so ;# An internal driver
#ns_section ns/db/driver/postgres
#ns_param pgbin /usr/lib/postgresql/8.1/bin/psql
ns_section ns/db/pools
ns_param maindb Main database Pool
ns_param session Session database Pool
ns_section ns/db/pool/maindb
ns_param driver postgres
ns_param datasource $env(PGHOST):$env(PGPORT):$env(PGDBNAME)
ns_param user $env(PGUSER)
ns_param password
ns_param connections 20
ns_param logsqlerrors true ;# Verbose SQL query error logging
ns_param verbose false ;# Verbose error logging
ns_param maxidle 600 ;# Max time to keep idle db conn open
ns_param maxopen 3600 ;# Max time to keep active db conn open
ns_section ns/db/pool/session
ns_param driver postgres
ns_param datasource $env(PGHOST):$env(PGPORT):$env(PGDBNAME)
ns_param user $env(PGUSER)
ns_param password
ns_param connections 20
ns_param logsqlerrors true ;# Verbose SQL query error logging
ns_param verbose false ;# Verbose error logging
ns_param maxidle 600 ;# Max time to keep idle db conn open
ns_param maxopen 3600 ;# Max time to keep active db conn open
# Accessing DB pools
ns_section ns/server/${servername}/db
ns_param pools * ;# Wildcard gives access to all
ns_param defaultpool maindb
You can try on Ubuntu packages aolserver4 and aolserver4-nspostgres from my
repository:
deb http://mobigroup.ru/debian/ squeeze main non-free
deb-src http://mobigroup.ru/debian/ squeeze main non-free
--
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Git on SourceForge
On Wed, Nov 17, 2010 at 6:19 AM, Jade Rubick jrub...@truist.com wrote: The main argument for github is that it supports a public collaborative usage of git. Unless we hear otherwise, so far I think we can summarize this thread as: Tom strongly dislikes github. Several other people favor it. The rest don't care or haven't spoken up yet. I personally think github is the way to go. There is a reason it has dominated so many open source projects. The main benefit is visibility: you get visibility into everyone's repositories, while tracking what is the canonical repository. That seems highly valuable to me. Okay, it seems like sf's gitweb search isn't working, so my main reason for staying with sf has evaporated. So let me ask about ideas for future development. There are 64 projects on sf, but only about 5 on github's aolserver account. I assume that aolserver will be the main repo (for current sf code) and developers will copy what dossy has done and fork whatever projects are necessary for development. Then developers can add related projects (AOLserver modules and/or patches) to their account. Eventually some of this might get pulled into aolserver. I've got a script to pull all the projects out of sf's cvs, creating a transcript and moving the repos. I could add to the script so that a README is generated as well as some kind of description and maybe push this to github. I'll test it out at my rmadilo account on github. Comments? tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] Git on SourceForge
Hi, Since everyone here knows that I have no ability to be nice, let me git to the point. GitHub sucks! It sucks because it costs money. It sucks because it doesn't use gitweb. It sucks because of poor user/group management. It sucks because it really sucks...hard! It sucks because is uses a little too much javascript. It sucks because it fucks up search in general (see gitweb). Basically the interface isn't gitweb and is oriented towards making money. The alternative: sourceforge. I know that sourceforge sucks...if you can run your own server. The problem with that is how much do you trust me to keep my server up and running? I don't trust myself. Do you trust the money-grubbers at github? Basically they are Facebook for programmers: willing and primed to sell your IP out at every opportunity. An alternative? How about git on sourceforge? Unlike github sf costs nothing. Unlike github sf allows infinite subprojects which cost nothing. Unlike github sf allows infinite developers the ability to commit to any project. Most important: Unlike github sf doesn't require you to bend over at a certain angle and accept your fate. As a starting point consider the following repo: http://aolserver.git.sourceforge.net/git/gitweb-index.cgi In this repo we have all projects and every project has a complete commit history. How many projects: 64! I haven't looked into how much github would charge for 64 projects, but that is how many we have right now. Where are these projects on sourceforge? http://aolserver.git.sourceforge.net/git/gitweb-index.cgi Although I caution everyone that there is no community consensus on the use of this repository, please note that everyone who could commit to the sourceforge CVS repository can commit to this new GIT repository. I'll post relevant urls very soon. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Git on SourceForge
On Tue, Nov 16, 2010 at 6:18 PM, Jeff Davis da...@xarg.net wrote: On 16 Nov 2010, at 18:15, Tom Jackson wrote: http://aolserver.git.sourceforge.net/git/gitweb-index.cgi I am not sure why you think using gitweb is good - I find it pretty unpleasant. I pointed this out months ago. gitweb is just a cgi program. But it has a very good search feature. When you search on github, you search every repo. Do you think this is helpful? When you search with gitweb you search below your current hierarchy, plus the pickax feature is invaluable. I also pointed out that many github features are implemented as javascript/web forms. For people who like drop down menus as opposed to a list of hyperlinks, github must be wonderful. More to the point is that any clone has a full revision history so I am not sure why you would really care so much about the web interface. I publish my repositories so that I can search them via gitweb! Publishing a repository has more than one purpose. With CVS, the purpose is somewhat limited. With gitweb, you can easily grab a snapshot of any subtree. Maybe github has enabled subtree snapshots, but last time I looked you could only get a top level snapshot. I like the setup on github a lot better. Forking/watching is great and submissions of pull requests is a good way to handle dvcs integration. Forking makes a public copy of the original project. What is the point of this? Now every source code search will return hits for each fork. Totally messed up. github is free for open source projects; I regularly see people with 15+ forked repos and I suspect if you ask they will increase your storage limit for a large open source project particularly if its the reference repository. If I ask? I have to beg github? Forks of a large project don't seem to go against your allocation as well (although maybe they do - need to confirm that). I forked postgres and my dashboard did not show my space used as any larger. Point is we know what sf offers. Yeah they have a business model but I fail to see the anal violation inherent in that. The whole point of the business model is to disguise the violation. Apparently they are doing a good job. tom jackson BTW, there are five projects at github out of 64. On sf we have 64/64 and all previous permissions are maintained. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Git on SourceForge
Truth is: who cares? Unless you want a canonical version of AOLserver. But that argues against the github model which creates a fork for every developer. Is it possible that I could maintain the sf version of AOLserver which allows multiple developers to maintain private repositories and commit changes to sf as needed? Or does the github model require total submission? tom jackson On Tue, Nov 16, 2010 at 5:37 PM, Jade Rubick jrub...@truist.com wrote: I personally strongly favor github. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Has anyone built nsd3.3+ad13 on 64-bit Linux?
Janine, My guess is that the code was run inside a namespace instead of at the global level. This makes the proc name relative to that namespace. The other possibility is that the proc was sourced inside a namespace and became part of that namespace. tom jackson On Sat, Nov 6, 2010 at 9:24 AM, Janine Ohmer jan...@furfly.net wrote: Thanks, Gustaf. I finally abandoned this and got it working (as far as I can tell, anyway) under 4.0.10. janine On Nov 6, 2010, at 3:02 AM, Gustaf Neumann wrote: If you want to compile the vintage version, i would will have to compile either with -m32 or to backport the 64bit fixes of the last 10 years, which are most likely necessary for aolserver33 as well. Search e.g. through http://www.ohloh.net/p/aolserver/commits?query=64-bitcommit=Update http://www.ohloh.net/p/aolserver/commits?query=64-bitcommit=Update and similar search patterns best regards -gustaf neumann On 05.11.10 23:03, Janine Ohmer wrote: Followup to this... The original build error was: (cd /usr/local/src/aol33+ad13/aolserver/tcl7.6/unix; make CFLAGS='-I../include -D_REENTRANT=1 -DNDEBUG=1 -g -fPIC -Wall -Wno-unused -mcpu=x86_64 -DHAVE_CMMSG=1 -DUSE_FIONREAD=1 -DHAVE_COND_EINTR=1' libtcl7.6.a) make[2]: Entering directory `/usr/local/src/aol33+ad13/aolserver/tcl7.6/unix' gcc -c -I../include -D_REENTRANT=1 -DNDEBUG=1 -g -fPIC -Wall -Wno-unused -mcpu=x86_64 -DHAVE_CMMSG=1 -DUSE_FIONREAD=1 -DHAVE_COND_EINTR=1 -I./../generic -I. -DHAVE_UNISTD_H=1 -DHAVE_SYS_TIME_H=1 -DTIME_WITH_SYS_TIME=1 -DHAVE_TM_ZONE=1 -DHAVE_TM_GMTOFF=1 -DHAVE_TIMEZONE_VAR=1 -DSTDC_HEADERS=1 -DNEED_MATHERR=1 -DHAVE_SYS_IOCTL_H=1 -DTCL_SHLIB_EXT=\.so\ ./../generic/panic.c `-mcpu=' is deprecated. Use `-mtune=' or '-march=' instead. ./../generic/panic.c:1: error: bad value (x86_64) for -mtune= switch I fixed this with: # CFLAGS+=$(GCCOPT) -mcpu=$(shell uname -m) CFLAGS+=$(GCCOPT) -m64 -mtune=generic in aolserver/include/Makefile.global. Now everything builds, but it starts to run and then seg faults. *sigh* I also tried CFLAGS+=$(GCCOPT) -O2 -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic (everything from the new Makefile.global that the old one doesn't have) But it still seg faults. janine On Nov 5, 2010, at 1:58 PM, Janine Ohmer wrote: I just posted the following to at openacs.org, and wanted to see if anyone who lurks here but not there might have any ideas: Due to unforeseen circumstances (server death :) I'm moving some very old sites to a new system running 64 bit Redhat (old system was 32 bit). A couple of these sites were running under nsd3.3+ad13. To my great surprise most of nsd worked without having to be rebuilt, but the postgres driver did not work and I was unsuccessful at building a new one. Not wanting to invest a lot of time on an ancient webserver, I went ahead with the move and set the 3.2.5 sites up under AOLserver 4.0.10, also quite old but the client didn't want me to upgrade anything more than was absolutely necessary. Unfortunately, they don't work. There are no errors in the error log on startup but the page load dies with: invalid command name template::adp_parse There don't seem to be many people reporting this error. I've seen this thread: http://openacs.org/forums/message-view?message_id=26808 But have double-checked the config file (several times!) and Home is set correctly. However, this makes me think that the problem is probably due to changes to AOLserver's module/tcl scripts. So does anyone know how to either a) run 3.2.5 under AOLserver 4 or b) get nsd3.3+ad13 built and working on 64 bit Linux? Thanks in advance! --- Janine Ohmer (formerly Sisk) President/CEO of furfly, LLC 503-693-6407 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email tolists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. --- Janine Ohmer (formerly Sisk) President/CEO of furfly, LLC 503-693-6407 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email tolists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from
Re: [AOLSERVER] Unable to bind to port 80
Possible that the daemontools setup is not correct? I think I was the first person to use daemontools with AOLserver, maybe check out: http://rmadilo.com/files/aolserver-daemontools/ tom jackson On Fri, Nov 5, 2010 at 9:33 AM, Janine Ohmer jan...@furfly.net wrote: I'm installing a group of old OpenACS sites on a managed server at Rackspace and have run into a problem I just can't seem to figure out. The two sites that need to run on port 80 are unable to bind to it - I get permission denied errors. I've checked all the usual things - the sites are being started by daemontools, so the prebind is done by the root user, and I do have the -b ip:80 in both run scripts. Judging from Google, those are the two most common errors. I'm using nsd 4.0.10; I know that's ancient history but the client didn't want me to upgrade anything and it was working fine on the old system. This is the first time I've used it on a 64 bit system (Redhat), for what that's worth. I did build it from scratch on the new system, of course. When I first started working on this Apache had a Listen 80 statement in the config file and I thought that was the culprit, but it has now been changed to specify its own (different) IP address and restarted several times. So that's not it. I don't see anything wrong in my setup and I also don't see anything suspicious in netstat that looks like the port is in use. I've asked the Rackspace folks to look at the system and confirm for me that nothing else has grabbed the port, but I'm sure not seeing it. Are there other gotchas I can check for, or known problems I'm unaware of? thanks, janine --- Janine Ohmer (formerly Sisk) President/CEO of furfly, LLC 503-693-6407 -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] WebSockets in AOLserver: get the full request body
On Mon, Nov 1, 2010 at 1:23 AM, w...@dcon.at wrote: So it's simply not possible to retrieve the 8 bytes after the header? That's a pitty, because that would be all I need for implementing the WebSocket protocol and with ns_chan one could feed data to a lot of clients. One thing you will have to handle is the Upgrade header field. This changes HTTP/1.1 to websockets (from the old draft): 1.3. Opening handshake _This section is non-normative._ The opening handshake is intended to be compatible with HTTP-based server-side software, so that a single port can be used by both HTTP clients talking to that server and WebSocket clients talking to that server. To this end, the WebSocket client's handshake appears to HTTP servers to be a regular GET request with an Upgrade offer: GET / HTTP/1.1 Upgrade: WebSocket Connection: Upgrade So it might be possible. One way to do it might be to hand off the connection to a Tcl thread (currently there is a patch and/or a module for creating background delivery of large content using [fcopy]. However there are other problems which conflict with HTTP: the headers are a mix of unicode and UTF-8, which doesn't match up with how HTTP headers are parsed. Not sure why the draft RFC authors think this is compatible with HTTP. The current draft RFC is here: http://datatracker.ietf.org/doc/draft-abarth-websocket-handshake/ which looks pretty much like a work in progress. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] WebSockets in AOLserver: get the full request body
I have an example of using the background module. I think it is compatible with the 4.5.1 patch. The main difference is that my version of the C code is a module with additional core API added to nsd. The Tcl code should work the same. Here is the link (the init.tcl shows how to use the thread package): http://junom.com/gitweb/gitweb.perl?p=aolserver.git;a=tree;f=aolserver/nsbgwrite;hb=HEAD I'll look at the patch. It seems useful even for more than the current issue. Also, it might be possible to use the new registered filter point: prequeue to completely avoid using a conn thread. tom jackson On Mon, Nov 1, 2010 at 3:51 PM, Jeff Rogers dv...@diphi.com wrote: It looks like my previous answer was, at best, partially right. It looks lke [ns_conn channel] does in fact give you a read/write channel. The effect of the missing content-length header is to discard any data that was read past the headers. Attached is a patch (readahead.patch) to change that behavior. The patch does 2 things: one, if the request includes 'upgrade' in the connection header, it doesn't read past the header (and doesn't throw away what it has read). Two, when [ns_conn channel] is called, it will insert into the channel buffer any data that was already read (and not processed) on the socket. Combined, it means that on an upgrade connection you can get the entire body, even if there was no content-length header sent. Also attached is a tcl file (index.tcl) that looks for an upgrade header, and when it gets one it forks a background tcl thread that echoes everything read back to the sender. Every new request will get a new thread. It also probably needs some tweaking to get a second request to work at all (since it's using shared channels to give the socket to the background thread). Importantly, this design of thread-per-connection is not scalable - your server will fall down and start crying once you start up a few dozen threads. (perhaps more; a really rough guess is that each tcl thread consumes 2M of memory before doing anything). What you really want to do is to transfer each new websocket to a single existing background thread, but I'm not sure how to send stuff into another thread's event loop using the aolserver thread tools. The standard tcl thread package is more powerful (with thread::send and thread::wait) but I don't know if those play nice inside AOLserver. -J wiwo wrote: Hi! I just tried the following: I've been telneting to our server and was feeding it this request: GET /test/ns_chan HTTP/1.1 Host: www.dcon.at Connection: Upgrade Sec-WebSocket-Key2: 12998 5 Y3 1 .P00 Sec-WebSocket-Protocol: sample Content-Length: 8 Upgrade: WebSocket Sec-WebSocket-Key1: 4 @1 46546xW%0l 1 5 Origin: http://example.com ^n:ds[4U Notice the Content-Length header. As Jeff told me, I've now been able to output the 8 byte body of the request. Then I was creating an ns_chan with: set ch [ns_conn channel] ns_chan create $ch $chan_name and getting it back with: set ch [ns_chan get $chan_name] The I tried to write my WebSocket server handshake with: puts $ch HTTP/1.1 101 WebSocket Protocol Handshake Upgrade: WebSocket Connection: Upgrade Sec-WebSocket-Origin: http://[util::host] Sec-WebSocket-Location: ws://[util::host]/[ns_conn url] Sec-WebSocket-Protocol: sample $ret which showed up in the telnet client. Then I tried to read again set str [read $x 10] ns_log notice $str and typed 10 characters in the telnet client which showed up in the log. So far so good! I could now write my own proposal for changing the WebSocket Client and make Google and Apple change their implementations of the protocol. Unfortunately my relationship with Steve and Larry cooled down the last few months, so this could take a while :-) Or I could get my hands dirty and mess around with the code in driver.c. As far as I can see, the crucial code is in static ReadErr SockReadLine(Driver *drvPtr, Ns_Sock *sock, Conn *connPtr) somewhere at line 1878 where the content length is read. I could simply check for the Upgrade: WebSocket header and set the length to the magic number of 8. Of course I have no idea when I'm talking about AOLserver C code, but maybe someone with a clue can give me his/her thoughts here. wiwo I'm pretty sure it's not possible to implement websockets in AOLserver without some C-level work, and if you could you wouldn't want to. You can't, because the driver reads the entire request, including content, before handing it off to the connection thread. And it only reads past the headers if there is a 'Content-length' header. (You can hook into the content-reading with a read filter, but that won't help because with no content-length it won't ever try to read it). You wouldn't want to, because you would attach a conn thread to a single persistent websocket, which would severely limit how many you could handle. Doing
[AOLSERVER] Git now available on SourceForge.net
Hey, I just noticed that Git is available on SF and it seems to support the necessary features not available on github: Gitweb configured rationally, Multiple repos (to support modules as separate projects) SSH access for developers Multiple developers can commit to the same repo. Instant snapshot downloads of any revision. http://sourceforge.net/apps/trac/sourceforge/wiki/Git What I don't know yet is if the import from CVS transfers all the history. The biggest benefit of git is the easy ability to view and search all history. Pickaxe is amazing for finding old code. I also noticed that sourceforge's git wiki uses trak, and it is available as a hosted app. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLServer Terminal Escape Sequence in Logs Command Injection Vulnerability
After looking at the URI RFCs (twilight zone), here is a basic algorithm which should be applied to the request: 1: Unreserved chars should always be unescaped (if they are escaped), producing a normalized request uri (assuming no #3 illegal chars, this is what will go into the log file): unreserved = ALPHA / DIGIT / - / . / _ / ~ 2: reserved chars should be left as-is: reserved= gen-delims / sub-delims gen-delims = : / / / ? / # / [ / ] / @ sub-delims = ! / $ / / ' / ( / ) / * / + / , / ; / = 3: Any other char should be considered an exploit or client error. If the url is logged, it might end with the % encoded representation of the first illegal char, plus a response code to identify the client error. Since this decoding/normalization step should take place when processing the start-line/request-line, this might be a good time to update AOLserver to handle absolute uri, which will also require changes to the logic used for (virtual) host identification. The RFC support for this is from 3986: 2.1. Percent-Encoding A percent-encoding mechanism is used to represent a data octet in a component when that octet's corresponding character is outside the allowed set or is being used as a delimiter of, or within, the component. A percent-encoded octet is encoded as a character triplet, consisting of the percent character % followed by the two hexadecimal digits representing that octet's numeric value. For example, %20 is the percent-encoding for the binary octet 0010 (ABNF: %x20), which in US-ASCII corresponds to the space character (SP). Section 2.4 describes when percent-encoding and decoding is applied. pct-encoded = % HEXDIG HEXDIG The uppercase hexadecimal digits 'A' through 'F' are equivalent to the lowercase digits 'a' through 'f', respectively. If two URIs differ only in the case of hexadecimal digits used in percent-encoded octets, they are equivalent. For consistency, URI producers and normalizers should use uppercase hexadecimal digits for all percent- encodings. Anyway, it is critical to examine and normalize the request uri asap and act quickly when presented with invalid chars. tom jackson On Sat, Sep 11, 2010 at 4:00 PM, Tom Jackson t...@rmadilo.com wrote: This is not an AOLserver issue to write a log file that is safe for broken programs. If there are illegal chars in the URL, maybe reject the request outright. If the chars are legal, then there isn't much else to be done. Chances are the chars in question should be escaped in the URL, so the request should be rejected. Although it might be nice to inform the client, it might be okay and more safe to just drop the request with no response. tom jackson On Fri, Sep 10, 2010 at 9:34 AM, Dossy Shiobara do...@panoptic.com wrote: Fair enough. ;-) On 9/10/10 2:07 AM, Gustaf Neumann wrote: The information loss (changing ESC to the bell character 7) is very little; under normal operation, you should never have a bell character in the log file, and now, if you see one, it should ring a bell -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLServer Terminal Escape Sequence in Logs Command Injection Vulnerability
Dossy, I agree that the problem is not with AOLserver, wrt reading the log file, but AOLserver should probably not accept url bytes outside of the accepted range. Headers are slightly more difficult to handle. I have a test http client (htclient) which only accepts context appropriate bytes. For headers the range of acceptable bytes is larger, but you first have to identify the context. From what I can figure out right now, URLs can't contain unescaped control characters, so valid urls written to a log file will never cause problems with shell programs. On Mon, Sep 13, 2010 at 2:16 PM, Dossy Shiobara do...@panoptic.com wrote: Actually, someone made the point -- what if you log request *headers* and someone puts a malicious byte sequence in that header? What's the rule around escaping the header values? What about every other code path where a remote user can write unfiltered bytes to a file on the server (logfile, etc.). Essentially, the vulnerability here isn't in applications that write these bytes to files, but specific terminal applications that are weak and should be fixed. Otherwise, cat is potentially vulnerable and that's a ridiculous position to hold. This is absolutely correct. External applications should protect themselves against malicious input. Many do. Occasionally I make a mistake and read a binary file. The result is lots of beeps and strange graphical characters appearing on the terminal. The only solution is to exit the terminal. Headers are much more dangerous than the request-line, but the problem is with the reader, not the http application. The request line cannot contain malicious characters, I'll have to review the headers definition (either don't log headers or read them with a safe reader). tom jackson On 9/13/10 4:46 PM, Tom Jackson wrote: Anyway, it is critical to examine and normalize the request uri asap and act quickly when presented with invalid chars. -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLServer Terminal Escape Sequence in Logs Command Injection Vulnerability
This is not an AOLserver issue to write a log file that is safe for broken programs. If there are illegal chars in the URL, maybe reject the request outright. If the chars are legal, then there isn't much else to be done. Chances are the chars in question should be escaped in the URL, so the request should be rejected. Although it might be nice to inform the client, it might be okay and more safe to just drop the request with no response. tom jackson On Fri, Sep 10, 2010 at 9:34 AM, Dossy Shiobara do...@panoptic.com wrote: Fair enough. ;-) On 9/10/10 2:07 AM, Gustaf Neumann wrote: The information loss (changing ESC to the bell character 7) is very little; under normal operation, you should never have a bell character in the log file, and now, if you see one, it should ring a bell -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] About WebSockets support
I would look at the nsbgwrite: http://junom.com/gitweb/gitweb.perl?p=aolserver.git;a=tree;f=aolserver/nsbgwrite This is also in the AOLserver head. Basically, pass off a socket to a dedicated thread and use the tcl non-blocking i/o to multiplex communication. If you want to actually interact with the server, you could look at my very deficient nsjabber code: http://rmadilo.com/files/nsjabber/ Jabber maintains persistent connections to clients, so this code is an example of what you are trying to do. Of course jabber sucks and requires xml events, but this is what you get with persistent connections. On Tue, Mar 23, 2010 at 11:06 AM, Alexey Pechnikov pechni...@mobigroup.ru wrote: Hello! Is it posiible for AOL architecture to manage a lot of persistent connections? Or may be we need search another servers for this task? Best regards, Alexey Pechnikov. http://pechnikov.tel/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver on GitHub
On Sat, Apr 10, 2010 at 4:28 PM, Dossy Shiobara do...@panoptic.com wrote: OK, lets consider that a +1 vote. I'll throw in my own +1 vote. Lets open the discussion for ~1 month - on May 15th, we'll do a final tally. Does anyone have any concerns? Reasons why this change shouldn't be made? An alternate change they'd like to propose and vote on? I signed up for an account under rmadilo, but I have no idea how to join/collaborate on the aolserver project. How does github work with more than one person? tom jackson Also, here is how I CVSimport AOLserver into git: #!/bin/bash CVSROOT=:pserver:anonym...@aolserver.cvs.sourceforge.net:/cvsroot/aolserver MODULE=aolserver REMOTE= export CVSROOT # -i import only # -v verbose git-cvsimport -v -i \ -d $CVSROOT \ -p x \ -C $MODULE \ $MODULE -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver on GitHub
I'm perfectly fine with git, but I'm highly biased, given the fact that I've been using it for any AOLserver changes over a number of years. I would like to see the people who commit/maintain code agree, and that usually isn't me. I'm not very familiar with github however. When I compare it to gitweb, it kind of sucks. The searches don't include line number links. Maybe this can be configured. The interface looks more geared toward advertising than browsing code. This isn't critical for me I would likely pull in changes and use gitweb. I also don't see everything, I think the toplevel directory is missing, including aolserver.com and modules. Compare to this: http://junom.com/gitweb/gitweb.perl?p=aolserver.git;a=tree tom jackson On Sat, Apr 10, 2010 at 4:28 PM, Dossy Shiobara do...@panoptic.com wrote: OK, lets consider that a +1 vote. I'll throw in my own +1 vote. Lets open the discussion for ~1 month - on May 15th, we'll do a final tally. Does anyone have any concerns? Reasons why this change shouldn't be made? An alternate change they'd like to propose and vote on? On 4/10/10 9:35 AM, Jade Rubick wrote: Yes, let's officially move it to github. Awesome change! -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver on GitHub
Over at http://junom.com/gitweb/gitweb.perl?p=aolserver.git I've sucked out all commits ever to CVS. Main use of this is the grep and pickax searches. Are these available on github? Oh, and the instant snapshot downloads (zip or tgz format). tom jackson BTW, what script do you use to keep the git repo up-to-date with CVS? On Fri, Apr 9, 2010 at 9:19 PM, Dossy Shiobara do...@panoptic.com wrote: Hi, So, I've finally gone crazy and have started pushing the contents of the CVS repository from SourceForge up to GitHub: http://aolserver.github.com/ http://github.com/aolserver I'm hoping this will make it easier for people to fork the code, make their own individual changes, and share those changes with the rest of the community. I'm hoping this will effectively eliminate any previous barriers to contibution - fork the code on github as much as you'd like in your own account, share your changes with whoever you please. Is this an official change? Of course not! The code still lives in CVS at SourceForge, like it always has - but now a copy with full CVS history is up at GitHub, as well. At this point, what does it mean to be official anyway, right? As of right now, I've only imported the aolserver module from CVS into GitHub. Over the next several weeks, I plan to import all the modules from CVS, but this takes time. A nice feature of GitHub is the included Issues system, their Wiki, and their static page hosting. If I get really ambitious, I might look to move as much of the AOLserver documentation as I can into the GitHub Pages area. As usual, feel free to flame me for running off into the weeds and just doing something without getting consensus or involving the community but I'm hoping at least a few of you will find this work worthwhile. If you have any positive comments and/or suggestions, don't hesitate to get in touch with me: I'd love to hear what you think. AOLserver's not dead, yet. ;-) -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] how to parse Content-Type text/xml form post
Here's an example of an adapter which allows you to develop your xml
code independently of your AOLserver code:
proc ::wsdl::server::accept { why } {
log Notice Accepting Connection with $why
foreach {server service port binding address} $why {}
# 0. Get POSTed Data:
set headerSet [ns_conn headers]
set length [ns_set iget $headerSet Content-length]
set tmpFile [ns_tmpnam]
log Notice Using tmpFile = $tmpFile
set fp [ns_openexcl $tmpFile]
fconfigure $fp -translation binary
ns_conn copy 0 $length $fp
close $fp
# 0.1 Package File and Headers:
set headerLength [ns_set size $headerSet]
for {set i 0} {$i $headerLength} {incr i} {
lappend headers [ns_set key $headerSet $i] [ns_set value $headerSet $i]
}
set requestID [::request::new $headers $tmpFile $why]
# Note Binding
log Notice wsdl::server::accept binding = $binding
# 2. Let Binding handle the request
set responseList [[set
::wsdb::bindings::${binding}::handleRequest] $requestID]
ns_return [lindex $responseList 0] [lindex $responseList 1]
[lindex $responseList 3]
}
Instead of ns_conn content, I use ns_conn copy to create a temporary
file for the xml document. The filename and a tcl copy of the headers
and request is passed to a potentially AOLserver ignorant package.
tom jackson
On Wed, Apr 7, 2010 at 7:09 AM, Brad Chick b...@doitsports.com wrote:
Thanks. That was simple enough.
Just a ns_conn content and then tdom.
It just wasn't easy to find much info on this.
On 4/7/2010 4:31 AM, Gustaf Neumann wrote:
Is there something wrong with ns_conn content ?
Am 07.04.10 09:03, schrieb Bas Scheffers:
I think the question is *how* to grab the raw post data?
On Wednesday, April 7, 2010 4:12pm, Gustaf
Neumannneum...@wu-wien.ac.at said:
Grab the post data and use an xml parser to get the content
There are multiple xml parsers for tcl available that you can use.
I would recommend tdom, which supports among other
things xpath (see http://www.tdom.org/)
-gustaf neumann
Am 07.04.10 06:40, schrieb Brad Chick:
This should be fairly simple, but I having issues parsing a form
post that is
of Content-Type text/xml.
The methods I see to handle the form data all try to parse the body
into
key/value pairs.
This is not what I need.
Rather, I just need the xml payload.
any help would be appreciated.
Thanks
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email
tolists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the
Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the
Subject: field
of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email
tolists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the
Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the
Subject: field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] installation problems with 4.5.1 on centos 5.4 64-bit
Make sure you are using the tclsh in /usr/local/aolserver/bin with library in /usr/local/aolserver/lib, according to the output below, your tclsh is in /usr/local/lib. with configure/make/make install, you need something like this: ./configure TCLSH=/path/to/tclsh ... make TCLSH=/path/to/tclsh I recommend that you compile your own tcl and put it in /usr/local/aolserver (as prefix) instead of using the local version (which might not have threads). tom jackson On Mon, Feb 1, 2010 at 9:21 AM, Brad Chick b...@doitsports.com wrote: i keep getting: ranlib: '/usr/local/aolserver/lib/libnsthread.a': No such file gmake[1]: *** [install-dll] Error 1 gmake[1]: Leaving directory `/home/brad/aolserver-4.5.1/nsthread' gmake: *** [install-bins] Error 1 I have tried with both tcl 8.4 and 8.5, but keep getting that same error. I made sure I used enable-threads and had the install location right: ./configure --prefix=/usr/local/aolserver --with-tcl=/usr/local/lib/ --enable- threads I also ran make clean on reintsalls. Any help would be great. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
On Fri, Jan 22, 2010 at 3:03 PM, Jeff Rogers dv...@diphi.com wrote: The YUI upload control looks like a good place to start for the flash client-upload feature. I haven't looked into it too deeply tho, so I don't know what the server side looks like. YUI Uploader widget: http://developer.yahoo.com/yui/uploader/ Other that that, I was pondering the plain upload issue. Since IE/Chrome/Safari are timing out on the upload, I wonder if the connection could be kept alive by sending something - anything - back to the client while it is still uploading. This is just caused by a brain damaged application. TCP/IP handles connection timeouts all by itself. As long as packets are being sent and acknowledged received, the application should not care. But very likely what is happening is that you have a blocking worker thread with is being controlled by another thread just using a simple timeout, without monitoring progress. Anyone who has noticed their browser freeze while loading google analytics, or some other ad iframe has experienced this poor event programming model. Either Firefox avoids this with active monitoring, or it doesn't use a timeout at the application level, or the timeout is very large. This might be doable with Jim's new read filter. Of course, the browsers might respond to data by closing their connection or stopping sending, or crashing (you never know with IE). And then even if it works, you have the problem of not having the tcp connection interrupted for however long it takes, which can be iffy in the world of flaky wireless connections and ISPs. Until the entire POST is complete, you have no method of communicating back to the client, this is the ultimate cause of the no progress being reported. To stay within the HTTP protocol, you would have to send multiple smaller chunks, and wait for the server to acknowledge it has received the data at the application level. Also, the chunked transfer encoding doesn't really help here since proxies are sometimes required to remove this encoding, cache the entire body and maybe retransmit it in chunks. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
I don't have any problem with this solution. It is superior to using a
forward proxy which uploads the entire file then reports progress to
the final server (this was the original model proposed in this thread,
by example).
In fact, I pointed out that the server thread is a proxy, handling
upload prior to allocating a conn thread. If you could peek into this
process, you get feedback.
But this is extremely inefficient. All of these solutions require a
specialized client, even it it seems somewhat transparent to the end
user. So I have pointed out several times that the best solution is a
client side solution which tracks upload progress by knowing the total
file size and the amount of bytes sent.
This really only make sense when the client is somewhat smaller than
the typical upload. Also, downloads are usually faster than uploads,
so the specialized client looks more attractive. the only critical
factor is ease of installation of the client.
Given the size of a tcl client, about 2 meg, any website with a
typical upload of 3+ megs would benefit from an easy to install and
use specialized client. My guess is that the javascript, flash and
java clients could be smaller, but would vary more than a simple tcl
client, which would work unchanged at the script level.
tom jackson
On Mon, Jan 18, 2010 at 12:51 PM, Jim Davidson jgdavid...@mac.com wrote:
Hi,
I think we were talking about this about a month ago. I updated the source
to enable upload-progress checking with a combination of ns_register_filter
and nsv -- there's an example at the latest ns_register_filter man page
(pasted below). This may work for you although it would require compiling
from latest sources. It assumes you have some javascript thinger that makes
repeated calls to check the status of the upload in progress on another
thread.
-Jim
EXAMPLE
The following example uses a read filter to update status of
a large HTTP POST to the
/upload/key url where key is some client-specified unique value.
While the upload is in
progress, it can be monitored with repeated GET requests to the
/status/key url with the same
key:
#
# Register procs to receive uploads and check status
# mainted in an nsv array.
#
ns_register_proc POST /upload upload.post
ns_register_proc GET /status upload.status
proc upload.status {} {
set key [ns_conn urlv 1]
if {[catch {set status [nsv_get status $key]}]} {
set status unknown
}
ns_return 200 text/plain $status
}
proc upload.post {} {
set key [ns_conn urlv 1]
nsv_unset status $key
# ... do something with content ...
ns_return 200 text/plain received
}
#
# Register a read filter ot update status
#
ns_register_filter read POST /upload/* upload.update
proc upload.update {why} {
set key [ns_conn urlv 1]
set expected [ns_conn contentlength]
set received [ns_conn contentavail]
set status [list $expected $received]
nsv_set status $key $status
return filter_ok
}
On Jan 18, 2010, at 2:39 AM, John Buckman wrote:
On 11/24/09 5:13 PM, John Buckman wrote:
Is there any access (in C or Tcl) to an upload-in-progress in aolserver?
It'd be nice if we extended ns_info with [ns_info driver ...] that could
give you connection-level info. from the driver thread. In its simplest
form, all we need is to expose the total bytes read/written on a socket
from the driver thread. Bytes read of the POST request's body and the
anticipated Content-Length enables us to compute a rough progress -
using the unique URL bit gives us an opaque handle to identify which
connection we're interested in.
I've learned a few things by deploying a large-file-upload feature on
aolserver:
1) IE times out on large file uploads over DSL, as does Chrome and Safari.
Only Firefox seems to have a long enough timeout to enable 600mb file
uploads over DSL.
2) All the other file upload sites use a client-side widget to upload a file
in parts, not using the browser's upload feature at all. Then, they have a
thin server-side program which accepts small chunks of the file upload at a
time. Once the widget decides the entire file has been sent, it submits to a
new web page, which then collects all the small file chunks.
So... instead of working on an upload-in-progress feature, it would make
sense instead to have a client-side widget (javascript/flash/java) that
sends file upload chunks to a server-side tcl script, and then have a
harvester tcl script once the widget says the file upload
Re: [AOLSERVER] Is this vulnerable to sql injection?
On Sun, Dec 6, 2009 at 9:27 PM, Don Baccus dhog...@pacifier.com wrote: Just go away until you 1) understand the SQL standard and 2) Oracle. Don! I am not the author of a database driver which cannot distinguish the empty string and NULL, a driver for a database which can make the distinction. That is you. When dealing with pure SQL I prefer a model which does not allow NULL. However, you must realize that the interface which we are talking about, which includes bind variables, deals also with a functional interface. This is where Oracle and PostgreSQL differ very significantly. Oracle allows default values whereas pg does not. PG also allows overloading, but Oracle does not. (or maybe it does, but I've never used it) So the problem of creating a generic interface to Oracle and PG presents complications far beyond handling NULL. I prefer Oracle because for any given problem it offers more tools to construct an optimal solution, but I am willing to work with PG for less demanding applications. But the application should not be defined in terms of the capabilities of any particular programming language. That is the basic problem with arguing that Tcl cannot represent NULLs. The real question is can Tcl code reflect the full semantics of a particular interface. Handling NULLs is easier than handling the so-called object-relational interface. Whatever, tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Is this vulnerable to sql injection?
On Sat, Dec 5, 2009 at 6:01 PM, Don Baccus dhog...@pacifier.com wrote: On Dec 5, 2009, at 5:13 PM, Tom Jackson wrote: Tcl can write a query string which uses the keyword NULL. Unfortunately the simple (but very nice and also safe) bind variable concept doesn't handle this common requirement. You've got it backwards, it's the empty string, not NULL, which a bind variable in Oracle (or in our emulation for postgres), doesn't handle - it becomes NULL (this is true of the empty string in Oracle in general). The requirement is the ability to handle both the empty string and NULL. Personally I like the Oracle behavior. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Is this vulnerable to sql injection?
On Sat, Dec 5, 2009 at 8:37 AM, Don Baccus dhog...@pacifier.com wrote: On Dec 4, 2009, at 12:03 PM, Tom Jackson wrote: The problem remains for other databases using the ns_db APIs. The quoting examples are general, but don't always work. The person's using PG so a PG-specific solution solves the problem. No reason to make this more complex when a simple solution suffices. Really? The title of the post says AOLserver is vulnerable. The example code uses [ns_db]. In case anyone else is interested, you can avoid SQL injection without using bind variables, regardless of which database or driver you use. Another problem is working with nulls. You can't quote null and postgresql distinguishes the empty string from null. Tcl doesn't implement the null concept. 'set foo ' sets foo to the empty string, not null. Tcl can write a query string which uses the keyword NULL. Unfortunately the simple (but very nice and also safe) bind variable concept doesn't handle this common requirement. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Jeff, Interps are confined to a specific thread. You can transfer the sock around, but not the interp. But the big reason for different interps is that they are or can be specialized. The prequeue interp could be very simple. Conn interps tend to be big and expensive so you don't want to use them any more than you have to. This is why we have a driver thread handling all i/o prior to queuing the conn. The driver thread is the upload equivalent of the download helper. (plus a bunch of other stuff). As an example of an additional prequeue filter, maybe the new http client (written in C) using the ?ns_task? API. Basically seems like you could dole out a task and return once the task (getting something via event driven http) is complete. Maybe a good use of the prequeue filter would be to actually return this download progress information. Since it is a filter, it would fire on a particular url. Maybe you could do a quick return and abort the connection, never using a conn thread. (Note this is a separate http request from the upload request.) It should be very fast if you never fire up an interp. tom jackson On Tue, Dec 1, 2009 at 1:45 PM, Jeff Rogers dv...@diphi.com wrote: Jim Davidson wrote: Right -- the pre-queue thing operates within the driver thread only, after all content is read, before it's dispatched to a connection. The idea is that you may want to use the API to fetch using event-style network I/O some other bit of context to attach to the connection using the connection specific storage API. So, it won't fire during the read. Can you share any specific examples of how it has been used? It's always struck me as an unfinished (or very specific-purpose) API since it's undocumented and it seems that doing anything nontrivial is liable to gum up the whole works since the driver is just a single thread. However, a progress callback API could be good -- wouldn't be called unless someone wrote something special and careful enough to not bog things down. Maybe an onstart, onend, and on % and/or on # callback which could do something like dump the progress in some shared (process or machine wide) thinger for other threads to check. I like the idea... Oh, and the pre-queue leak sounds like a bug -- should just be re-using the same Tcl interp for all callbacks. In the case of a tcl filter proc, ProcFilter gets the server/thread specific interpreter for the connection and expects NsFreeConnInterp to be called later, but in the case of pre-queue filters NsFreeConnInterp is never called in the driver thread so it allocates (PopInterp) a new interp every time. Adding in a call to NsFreeConnInterp after the prequeue filters looks like it fixes the problem. If a filter proc is added into SockRead the same thing would need to happen (potentially in the reader thread instead of the driver thread). One thing I am confused about tho, is why without calling NsFreeConnInterp in the driver thread it just leaks the interps rather than crashing when it tries to use the interp in the conn thread, since it looks like a new conn interp wouldn't get allocated in that case. I also don't understand why there can be multiple interps per server+thread combo in the first place (PopInterp/PushInterp); I'd expect that only one conn can be in a thread at a time and that it always releases the interp when it leaves the thread. -J -Jim On Nov 25, 2009, at 5:46 PM, Jeff Rogers wrote: It looks like the pre-queue filters are run after the message body has been read, but before it is passed off to the Conn thread, so no help there. However it looks like it would not be hard to add in a new callback to the middle of the read loop, tho it's debatable if that's a good idea or not (for one, it would get called a *lot*). Curious about that tcl prequeue leak. I guess no one uses or cares about it, since the symptom is serious (more than just a really big memory leak, it crashes the server too), the cause is pretty obvious and the fix appears on the surface to be pretty obvious, although it does result in prequeue filters working differently from all the others, in particular that it would use a different interp from the rest of the request. -J Tom Jackson wrote: There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER
Re: [AOLSERVER] upload progress
On Tue, Dec 1, 2009 at 3:28 PM, Jim Davidson jgdavid...@mac.com wrote: On Dec 1, 2009, at 4:45 PM, Jeff Rogers wrote: I also don't understand why there can be multiple interps per server+thread combo in the first place (PopInterp/PushInterp); I'd expect that only one conn can be in a thread at a time and that it always releases the interp when it leaves the thread. I think there was an edge case that led to a need for a cache of possible interps instead of just holding one. In practice, it's always just one unless someone writes some weird code to do a pop/push directly to have some alternate interp for a special purpose. One thing which causes multiple interps per thread is the default threadpool. If you have a default threadpool the threads are shared across servers. So, as each server uses a thread, an interp for that server is created. My advice is to always register a default threadpool for each server so that the process wide default threadpool is never used. I'm not sure if there is any security problem, but a shared threadpool makes each thread more expensive, and I doubt it saves resources. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
John,
I think your analysis is correct. You have to avoid certain API or
you do extra work, and maybe take up extra space. Usually this is
unimportant, but in your case it isn't.
Anyway, sounds like you are making progress. Please let me know if I
have led you down the wrong path. This stuff isn't really documented
anywhere.
tom jackson
On Tue, Nov 24, 2009 at 2:44 PM, John Buckman j...@bookmooch.com wrote:
Tom, I figured out the problem with the memory bloat when I used your
alternative ns_getform to parse a large file upload temp file.
The problem is that if I call ns_queryget after calling your ns_getform, that
causes aolserver to re-parse the large file upload, and to do it in the old
memory-inefficient way.
I'm not sure if there's a way for your version of ns_getform to tell the
aolserver internals that the parsing of the form is done, so subsequent
ns_queryget calls don't cause a re-parse.
At any rate, there's an easy workaround, which is to read things right out
the ns_set that your ns_getform populates.
-john
On Nov 23, 2009, at 8:47 PM, John Buckman wrote:
Tom, there is some sort of weird interaction effect on aolserver when doing
tcl stuff and there is a large file as the temporary file. Some cases cause
the nsd process to bloat to the size of the memory. I wasn't able to figure
out why. I wasn't able to fix your ns_getform to not bloat.
However, I was able to copy Vlad's ns_parseformfile proc for use on
aolserver from naviserver, and it doesn't bloat.
Here is the code for handling a large file uploaded as a file, rather than
in memory:
set type [ns_set iget [ns_conn headers] content-type]
set form [ns_set create]
ns_parseformfile $form $type
array set formdata [ns_set array $form]
puts final array: [array get formdata]
proc ns_parseformfile { form contentType } {
set fp [ns_conn contentchannel]
if { ![regexp -nocase {boundary=(.*)$} $contentType 1 b] } {
puts Warning: no MIME boundary
return
}
fconfigure $fp -encoding binary -translation binary
set boundary --$b
while { ![eof $fp] } {
# skip past the next boundary line
if { ![string match $boundary* [string trim [gets $fp]]] } {
continue
}
# fetch the disposition line and field name
set disposition [string trim [gets $fp]]
if { ![string length $disposition] } {
break
}
set disposition [split $disposition \;]
set name [string trim [lindex [split [lindex $disposition 1] =] 1] \]
# fetch and save any field headers (usually just content-type for
files)
while { ![eof $fp] } {
set line [string trim [gets $fp]]
if { ![string length $line] } {
break
}
set header [split $line :]
set key [string tolower [string trim [lindex $header 0]]]
set value [string trim [lindex $header 1]]
ns_set put $form $name.$key $value
}
if { [llength $disposition] == 3 } {
# uploaded file -- save the original filename as the value
set filename [string trim [lindex [split [lindex $disposition 2]
=] 1] \]
ns_set put $form $name $filename
# read lines of data until another boundary is found
set start [tell $fp]
set end $start
while { ![eof $fp] } {
if { [string match $boundary* [string trim [gets $fp]]] } {
break
}
set end [tell $fp]
}
set length [expr $end - $start - 2]
# create a temp file for the content, which will be deleted
# when the connection close. ns_openexcl can fail, hence why
# we keep spinning
set tmp
while { $tmp == } {
set tmpfile [ns_tmpnam]
set tmp [ns_openexcl $tmpfile]
}
catch {fconfigure $tmp -encoding binary -translation binary}
if { $length 0 } {
seek $fp $start
ns_cpfp $fp $tmp $length
}
close $tmp
seek $fp $end
ns_set put $form $name.tmpfile $tmpfile
if { [ns_conn isconnected] } {
ns_atclose ns_unlink -nocomplain $tmpfile
}
} else {
# ordinary field - read lines until next boundary
set first 1
set value
set start [tell $fp]
while { [gets $fp line] = 0 } {
set line [string trimright $line \r]
if { [string match $boundary* $line] } {
break
}
if { $first } {
set first 0
} else {
append value \n
}
append value $line
set start [tell $fp]
}
seek $fp
Re: [AOLSERVER] upload progress
John, I'm just going to venture a guess. I hope that Jim D. or someone else more familiar with the internals will set me straight. The problem with upload progress monitoring is that uploads are finished before a conn thread is allocated. Uploads are done in the driver thread, or a worker thread before the conn thread is given control. One thing which could help is if AOLserver introduced chunked transfer encoding for uploads. But to report progress back to the user would require a different setup. You would need to monitor the progress of some other process/thread. AOLserver has chosen to stratify io. By stratify I mean the io handling is moved through a pipeline. The pipeline may transfer handling between threads but also enforces the direction of communication. There is no asynchronous io, each step in the pipeline is focused on one-way communication. For instance, the driver thread only reads, the conn threads only write. There is a kind of fiction that conn threads do any reading from the client. These threads only read from buffers or disk. The reason should be obvious: dos attacks. The driver thread is highly tuned to protect the expensive conn threads. In addition to that issue, the connio uses scatter/gather io, so the application gets a very imperfect view of the actual progress. Of course the io is much more efficient and progress reporting would require an independent observation and messaging system. A first step would be the ability to log upload progress to error.log. If you can't do that, reporting back to the client will be impossible (unless you think the logging system should operate with less available information than individual conn threads). There is one possibility. There is a pre-queue filter in AOLserver (run inside the driver thread). It works from the Tcl level, but creates a memory leak equal to the size of an interp, in other words a huge memory leak. However, maybe at the C level, you could create a handler which would do something interesting before returning control back to the driver thread and ultimately the conn thread. I'm not sure exactly when the pre-queue filters are activated, but if it is before reading the message body, it might be useful. tom jackson On Tue, Nov 24, 2009 at 2:13 PM, John Buckman j...@bookmooch.com wrote: Naviserver has a very nice feature that allows (via javascript) to show a user the percent upload progress of a file. I tried porting their progress.c file to aolserver, but it's a significant effort, as it depends on other changes naviserver has implemented to the aolserver code. However, I was wondering if there wasn't a fairly simple way to implement something similar on aolserver. What I'd need is a way to know how much data has been uploaded. The way naviserver does it, is by asking you to POST your upload to a unique URL, and then they provide a tcl command that returns how many bytes have been uploaded to that unique URL. A javascript regularly polls an ADP page that uses the tcl progress command to tell you the number of bytes uploaded. Is there any access (in C or Tcl) to an upload-in-progress in aolserver? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Gustaf, I've seen these working, although I'm never sure where exactly the magic happens. It looks like the ngix idea is to work as a proxy: It works because Nginx acts as an accelerator of an upstream server, storing uploaded POST content on disk, before transmitting it to the upstream server. Each individual POST upload request should contain a progress unique identifier. I wonder if the progress is from you to ngix, or from ngix to the final server. On Wed, Nov 25, 2009 at 4:07 AM, Gustaf Neumann neum...@wu.ac.at wrote: Tom Jackson schrieb: John, I'm just going to venture a guess. I hope that Jim D. or someone else more familiar with the internals will set me straight. The problem with upload progress monitoring is that uploads are finished before a conn thread is allocated. Uploads are done in the driver thread, or a worker thread before the conn thread is given control. Tom, the typical meachnism for upload progress bars is that separate (ajax) queries are used to query the state of an running upload, which is identified by some unique ID (e.g. X-Progress-ID in http://wiki.nginx.org/NginxHttpUploadProgressModule, or some other heuristics. e.g. based on URL and peer addr). So, one needs just a thread-safe introspection meachnism, which checks in the spooler the state (such as ns_upload_stats in naviserver) and returns it back. The spooler hast just to update these statistics when it recieves data. -gustaf neumann -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] upload progress
Gustaf, Oops, accidentally hit send. I just started work on an event driven http client (called htclient). It can monitor downloads just by using a variable trace. I haven't reversed the idea for uploads yet, but it would be easy. Not so easy is guessing the length of the encoded file prior to sending. Seems like a better solution for large file uploads would be to do the upload as binary data instead of multi-part and get direct local monitoring and the ability to cancel, and/or restart a failed upload. For clients who have tcl/tk installed, it would be easy to auto-generate a tcl script to handle one or several uploads. tom jackson On Wed, Nov 25, 2009 at 9:13 AM, Tom Jackson t...@rmadilo.com wrote: Gustaf, I've seen these working, although I'm never sure where exactly the magic happens. It looks like the ngix idea is to work as a proxy: It works because Nginx acts as an accelerator of an upstream server, storing uploaded POST content on disk, before transmitting it to the upstream server. Each individual POST upload request should contain a progress unique identifier. I wonder if the progress is from you to ngix, or from ngix to the final server. On Wed, Nov 25, 2009 at 4:07 AM, Gustaf Neumann neum...@wu.ac.at wrote: Tom Jackson schrieb: John, I'm just going to venture a guess. I hope that Jim D. or someone else more familiar with the internals will set me straight. The problem with upload progress monitoring is that uploads are finished before a conn thread is allocated. Uploads are done in the driver thread, or a worker thread before the conn thread is given control. Tom, the typical meachnism for upload progress bars is that separate (ajax) queries are used to query the state of an running upload, which is identified by some unique ID (e.g. X-Progress-ID in http://wiki.nginx.org/NginxHttpUploadProgressModule, or some other heuristics. e.g. based on URL and peer addr). So, one needs just a thread-safe introspection meachnism, which checks in the spooler the state (such as ns_upload_stats in naviserver) and returns it back. The spooler hast just to update these statistics when it recieves data. -gustaf neumann -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
John, I would ditch using ns_getform and roll your own instead. What you seem to have proven is that you can upload large files with AOLserver 4.5 and the overflow goes to disk. The question remains: how to deal with the data on disk? [ns_conn files] gives you offsets, how do you copy that data somewhere else without bloating memory? Maybe [ns_conn copy] is the problem. If it loads $len bytes into memory before it writes a file, that will bloat the memory. You might verify that this isn't a memory leak, but rather a high water mark on memory usage. Do two identical uploads, or three continue to increase memory usage, or does it stop growing? If it keeps growing, that could indicate some mem leak bug. tom jackson On Mon, Nov 23, 2009 at 6:18 PM, John Buckman j...@bookmooch.com wrote: Tom, thanks for the help! Setting maxinput as per: ns_section ns/server/$server1/module/nssock ns_param maxinput 1024 does indeed avoid memory bloat during the large file upload. I used your safer ns_getform: it works fine, However, your ns_getform causes the nsd process to grow to the size of the uploaded file. I can't figure out what it is in your code that uses lots of memory. Any idea? -john -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
On Fri, Nov 20, 2009 at 12:21 PM, John Buckman j...@bookmooch.com wrote: On Nov 19, 2009, at 8:56 AM, Tom Jackson wrote: There is a configuration setting which saves posted files to disk. You need the ns_limits maxupload to be large enough, then maxinput (not sure which config section) sets an upper limit for in memory data. Yes, the naviserver people talked about that, but the problem is that then you get the raw data, that you need to do something with, rather than temporary files. Are you sure? Check out NsConnContent (in driver.c) and Ns_ConnContent (in conn.c), they abstract the location of the data on disk or in memory. This should be transparent to the application. Ns_ConnContent just exports NsConnContent. Also note that streaming to disk is done in the driver prior to selecting a conn thread. The conn thread is where the content handler is located. I cannot imagine how you can short-circuit content handling. I'm currently trying out naviserver, as they do seem to have solved the large-file-upload problem that aolserver has. It's working for me, and nsd process size is staying at 50mb even with multiple gig file uploads being handled. Still not sure if you ever tested it on AOLserver 4.5. If this doesn't work, it would be a real surprise the only thing in memory should be the headers and file offsets (see ParseMultiInput). Are you using [ns_conn files]? tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] large file uploads
There is a configuration setting which saves posted files to disk.
You need the ns_limits maxupload to be large enough, then maxinput
(not sure which config section) sets an upper limit for in memory
data.
This gets decided in driver.c, around line 1892:
1891 max = connPtr-roff + connPtr-contentLength + 2;
1892 if (max connPtr-drvPtr-maxinput) {
1893 /*
1894 * Content will fit at end of request buffer.
1895 */
...
1899 } else {
1900 /*
1901 * Content must overflow to a temp file.
1902 */
1903
1904 connPtr-flags |= NS_CONN_FILECONTENT;
1905 connPtr-tfd = Ns_GetTemp();
1906 if (connPtr-tfd 0) {
1907 return E_FDAGAIN;
1908 }
tom jackson
On Tue, Nov 17, 2009 at 9:06 PM, John Buckman j...@bookmooch.com wrote:
I'm developing a music submission system for Magnatune.com using aolserver
and I'm seeing some problems with large file uploads.
Specifically, aolserver periodically crashes due to a malloc error.
It looks like aolserver stores files being uploaded (via form
enctype=multipart/form-data) in memory as they're being uploaded. When
people are uploading zip files of WAV-quality rips of CDs, that's 800mb of
RAM per upload. A few concurrent uploads, and splat.
It looks like Naviserver has changed the aolserver code in Naviserver 4.99.1
to
- Using temporary files for large uploads when content exceeds
pre-configured maxsize parameter
- New spooler thread for handling large file uploads
Does it make sense to port this naviserver code over to aolserver?
Or, is there another (easier) way to handle large file uploads that doesn't
use lots of memory?
-john
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] is it necessary
On Sun, Nov 1, 2009 at 6:52 PM, Bas Scheffers b...@scheffers.net wrote: On Saturday, October 31, 2009 10:30pm, Dave Bauer d...@thedesignexperience.org said: Of course you can say apt-get install aolserver4 already. Of course you can, but then what? Then you still have this completely alien server in front of you that you will need to configure. It puts people off. being just another language and API an Apache environment is going to be much more appealing to most users. The vast majority of people that *use* LAMP and not developers, nor have any heroic sysadmin skills. They just download WordPress or Drupal or Gallery, dump it into their page root and away they go. They may tweak a few templates with PHP code in them, but that's it. That's the kind of simplicity AOLserver needs to grow. We live in an Apache world; if you can't beat them, join them... I guess the question is: once you have made all these changes to AOLserver, what exactly do you have left? The answer is Apache running Tcl. Why not start with Tcl and run it under FastCGI? Which AOLserver API do you wish to continue to use after you add on Apache? I have an nsd emulation layer which replaces ns_conn, ns_return, etc, so that most tcl pages run as expected, I have another adapter which allows this layer to run inside of tclhttpd, which is similar to CGI. This allows my tcl pages to run inside AOLserver, tclhttpd, tcpserver, nstclsh, stand-along [socket] or [socket] with threads. It uses the same startup script in every case. What do I lose? Filters, modules, speed, ns_config. If we could somehow get modules and ns_config to work with nstclsh, it is relatively easy to replace the filter/registered proc functionality (already done that too) in Tcl code. So the only thing left is speed. I can wait a year and hardware improvements will solve that one by itself. But for me the big thing is the database API and connections. Only AOLserver provides that piece, so someone would have to demonstrate that modules would still work as they do in AOLserver before I could imagine this working. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] is it necessary
On Fri, Oct 30, 2009 at 9:36 PM, Bas Scheffers b...@scheffers.net wrote: I do this for a couple of sites, Apache's mod_proxy forwards stuff to AOLserver. This is mostly where we have a scarcity of IPs. But that still means you have to jump through hoops to install it, learn how to run it in production, etc. It is strange that the most valid reason to use Apache is a lack of IP addresses. But it's true. I'd love a being able to just say apt-get install apache2-aolserver apache2-aolserver-postgresql. As soon as apt-get blah-blah-blah starts installing software where I want it, with the options I want, I also would love it. But if I install an AOLserver for a client, they might want to keep their version for a few years, the next client might want the current version and a new postgresql. The apt-get stuff works exactly once per machine and it still doesn't give much flexibility. Eventually you have to learn how to do custom installs. I like the idea of a forward (caching) proxy, it fits in well with the filter type architecture in AOLserver. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] AOLserver templating options
On Mon, Oct 26, 2009 at 3:53 PM, Hossein Sharifi shar...@rateyourmusic.com wrote: 2) http://junom.com/document/twt/view/www/ - it seems to be actively maintained given the fact that there are files in that directory created last month, but is it easy to install? And has it been tested in a production environment? I'm the maintainer. The current home for this is the tnt project: http://www.junom.com/gitweb/gitweb.perl?p=tnt.git It is still the view package, but requires, at least, the tnt package for logging. If you need help installing, please take advantage and ask here or via direct email. Any helpful hits will be shared as better documentation. The templating system is called tic-tac-toe, somewhat shorter than A templating toolkit for tcl. Although the files you saw have been updated recently, the actual code hasn't changed. It is relatively stable and the extension mechanism is well defined, so only actual bugs demand attention. But the templating system is somewhat conservative. You have to add resources in order to access anything outside of a particular template (the setup tcl file adds the resources). The whole point was a safe system where templates of unknown intent could safely be used. So, by default, template cannot execute unsafe commands, but resources can execute arbitrary code. The Tic-Tac-Toe example ( http://junom.com/document/twt/view/www/tic-tac-toe.tcl ) is non-trivial and demonstrates many of the features. The tnt toolkit has hundreds of examples of template files and a few examples of how to organize a website wide templating system. The readme file explains the language basics: http://junom.com/document/twt/view/www/README The template compiler is written in flex/bison so it it fast, but it also validates syntax. When an error occurs, you get a temporary file which ends close to the error. It is usually easy to find and fix syntax errors. The compiled template is just tcl code, once compiled, the compiled version remains valid until the original template file gets a newer timestamp. (You can use touch to force a recompile). 3) Are there any others? Good question. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] is it necessary
You have to install AOLserver if you want to run OpenACS, but AOLserver is independent of OpenACS. But you might join the OpenACS community, and/or use their resources (openacs.org), they discuss a wider range of things than we do here. tom jackson On Thu, Oct 15, 2009 at 10:04 AM, nitin chandra nitinchand...@gmail.com wrote: Hi Everyone, Is it necessary to install OpenACS along with AoLserver? Nitin -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Bug in gzip compression module
Daniel, Without version info from the original poster, who knows? But for good measure, could you link to your fix so we can figure out if it has been undone? Your response indicates to me that you are more alert to the telltale signs of a particular defect. But this isn't something that I recognize. Help us out. thanks, tom jackson On Mon, Oct 12, 2009 at 11:32 AM, Daniel Stasinski m...@avenues.org wrote: Ugh, I've fixed this bug in AOLserver twice so far. Perhaps someone undid it again. Daniel | --- | Daniel P. Stasinski | http://www.saidsimple.com | dan...@avenues.org | http://www.disabilities-r-us.com | XMMP: m...@avenues.org | http://www.avenues.org | Google Talk: mooo | http://www.scriptkitties.com -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 3 aolserver questions reagrding calender/drop down list/security
On Tue, Sep 29, 2009 at 2:50 AM, Alexey Pechnikov
pechni...@mobigroup.ru wrote:
Hello!
On Tuesday 29 September 2009 08:11:36 Tom Jackson wrote:
How does '999'::int solve the problem?
The example return the equal result in different PostgreSQL versions
(may be result can depend from platform). Strict type cast is defined
in documentation and may be safety. And this is independant of the
database schema.
When you try to store 999 as is then database
can translate it to integere and return otherflow error, or translate it
to float or text without error. This depends of the database schema.
So code can't be validated without the schema of the database.
I'm not sure how this turned into a discussion of buffer overflows and
potential mismatch between programming language/database data types.
Unless you actually test your specific code with your specific
environment you will always face this possibility.
The issue with quoting user input (surrounding quotes) has to do with
detecting if the input could be a number or not. Some databases don't
allow numbers to be quoted, some do. If your database allows
surrounding quotes, you don't need to check if the user input looks
like a number, you can just add the surrounding quotes.
If you use a basic type system, you can do a little better than this.
For char and varchar, you can check the length of the user input
(after [string map {' ''} $string]). If an input has a default value,
this can be automatically added. Note that the default might be NULL,
which is not quoted, a type system could handle this situation easily.
If you need strict type checking in Tcl, try out the type code in
tWSDL. It has very good numeric type definition and validation
support.
I added a new comparison, which is helpful for those who rely on Tcl
to do type checking:
http://junom.com/document/twt/view/www/decimal3.tcl (code at decimal3.tcl~)
This compares xml schema's decimal type to Tcl int and Tcl double.
Tcl's double is very close, but does include hex and untrimmed
numbers. Like decimal2.tcl the canonical value is included. Tcl int
also fails to recognize some non-canonical integer values.
You can use tWSDL's type code to further restrict the numeric type.
For instance, the system uses the base type decimal to define other
types:
restrictDecimal xsd integer xsd::decimal {fractionDigits 0}
restrictDecimal xsd int xsd::decimal {fractionDigits 0 minInclusive
-2147483648 maxInclusive 2147483647}
restrictDecimal xsd nonPositiveInteger xsd::integer {maxInclusive 0}
restrictDecimal xsd negativeInteger xsd::integer {maxInclusive -1}
restrictDecimal xsd short xsd::integer {minInclusive -32768 maxInclusive 32767}
restrictDecimal xsd byte xsd::integer {minInclusive -128 maxInclusive 127}
You can see the type checking code which is generated from the above
commands here:
http://junom.com/ws/mywebservice/?ws=mywebservicens=::wsdb::types::xsd
You can test values against an example restricted decimal type defined as:
mywebservice::TestDecimal {minExclusive -321.01 maxInclusive 456.78
totalDigits 5 fractionDigits 2}
using this web form:
http://junom.com/ws/mywebservice/?op=TestDecimalValueOperationmode=display
tom jackson
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] 3 aolserver questions reagrding calender/drop down list/security
On Sat, Sep 26, 2009 at 2:23 AM, Alexey Pechnikov
pechni...@mobigroup.ru wrote:
Hello!
On Friday 25 September 2009 22:29:55 Tom Jackson wrote:
Personally I would use [string is double -strict] and quote anything
Tcl and PostgreSQL types are not equal.
tclsh8.5 [/tmp]string is double -strict 99
1
template1=# create temp table test(value integer);
CREATE TABLE
template1=# insert into test (9);
template1=# insert into test values (99);
ERROR: integer overflow
How does '999'::int solve the problem? (Hint: it
doesn't) If a value is numeric it doesn't need surrounding quotes, the
test [string is -strict double $string] is only used to check if the
value looks like a number, there is no way it can guarantee the number
is valid for the particular query (the integer may be an id, which
might need to satisfy a referential constraint, for instance).
So you send to PostgreSQL overflowed values without strict typing
which are potentially dangerous. Defaults type conversions in
PostgreSQL depends of version and so application behaviour is
undefined with other PostgreSQL version.
If you need strict type checking in Tcl, try out the type code in
tWSDL. It has very good numeric type definition and validation
support.
Basic examples:
http://www.junom.com/gitweb/gitweb.perl?p=twsdl.git;a=blob;f=packages/wsdl/ns/ns-xsd.tcl;h=1482a
Some comparisons with Tcl int:
http://junom.com/document/twt/view/www/decimal2.tcl
code:
http://junom.com/document/twt/view/www/decimal2.tcl~
Database and application can be placed on hosts with different
arch (32 and 64 bit). In this case you can't check integer type size
in application because database can have the different type size.
On Friday 25 September 2009 22:29:55 Tom Jackson wrote:
What
is better is a type system which handles each type in a separate API
and rejects unfound types.
The better is mapping in the driver and DBMS with dynamic typing. As
example for SQLite (the forms $value and :value are equal):
db eval {create table test(value int)}
db eval {insert into test (value) values (9)}
insert into test (value) values (99.99)
set value 1
db eval {insert into test (value) values ($value)}
set value {The test value with '}
db eval {insert into test (value) values (:value)}
sqlite select typeof(value),value from test;
real|1.0e+21
real|100.0
integer|1
text|The test value with '
And strict field typing can be released by constraints or triggers or
collations.
Dynamic typing? My database tables/columns don't change types. This
last example of yours is strange. Why would you have multiple
different types in one column? Why use a database at all?
tom jackson
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_odbc on Win32
Hopefully someone with a windows setup will take this up, or maybe a new module can be added without the need to merge with the old version. ODBC on windows is potentially much more useful than on *nix since everything it seems can be an ODBC datasource. Updating AOLserver to include a more complete ODBC/SQL CLI adapter would be a good long term goal for this community, the only real issue is that the current ns_db API works so well, or is easy enough to adapt to, it is not easy to justify the pain for the marginal gain. tom jackson On Thu, Sep 10, 2009 at 3:38 AM, Scott Goodwinsc...@scottg.net wrote: Joe did this work during last year's Tcl conference when he couldn't get ns_odbc to compile on his Windows laptop so he could use it during a tutorial. I'm certain he based the changes on the version of ns_odbc in CVS at that time, and I think the changes were mostly to the build process (Makefile). He did not add or delete any functionality; he just got it to compile with a newer Microsoft compiler. He sent me the changes but his message was promptly buried in the dog pile that is my inbox and I haven't even unpacked and looked at what he's done. Though I could try and merge the changes myself, I have zero Windows dev experience and no Windows systems at home or work to build and test it on (and no interest in moving in that direction). But I figured I'd make the changes available to the list so that others would have the opportunity to do the merge, build, test and commit for the rest of the community that runs AOLserver on Windows. /s. On Sep 10, 2009, at 4:43 AM, Tom Jackson wrote: I wonder what the changes were. This driver is missing several API: select and dml (it has exec which can handle both by itself, so maybe they were removed). I have a more complete driver for unixodbc which includes transaction support, and a shutdown callback. If the changes are not that difficult, maybe I could apply them to this version as well (probably a different file for windows). It may not be possible, I can see some internal API differences. Both drivers seem to be derived from a common source Here's the current version of nsunixodbc: http://www.junom.com/gitweb/gitweb.perl?p=nsunixodbc.git;a=tree;hb=18d7 Unfortunately I have nothing setup on windows. tom jackson On Wed, Sep 9, 2009 at 8:42 AM, Scott Goodwinsc...@scottg.net wrote: Joe Mistachkin did some work on ns_odbc (almost a year ago) to get it to compile with newer Microsoft compilers; I've attached the relevant files if someone with Windows experience wants to diff and merge the changes into the module and check-in to AOLserver CVS. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. /s. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Aolserver as multithreaded TCL soc ket server ?
I thought I sent this last week, but I used the wrong account, anyway: This example is not multi-threaded, but it handles multiple connections. The basic API are ns_socklisten, ns_sockaccept and ns_sockcheck. (http://rmadilo.com/files/nsapi/) The example also uses ns_socklistencallback, but the code is very cryptic. Basically the client contacts the server with a contact address/port then creates a server to listen for updates (server push). The server uses ns_socklistencallback but avoids hanging in a single thread because it just gets the contact information and starts a scheduled proc to handle the recurring request (push). The scheduled proc contacts the client on the given schedule and pushes the requested data. The client could actually handle requests from multiple servers by using ns_socklisten/accept: http://www.junom.com/gitweb/gitweb.perl?p=3Dtwt.git;a=3Dtree;f=3Dpackages/tclbean/tcl;h=3D8f0100 tom jackson On Tue, Aug 18, 2009 at 5:56 AM, Nikolay Shulganikolay.shu...@runway.lv w= rote: Hello, Is it possible to use Aolserver as multithreaded TCL socket server ? If I connect to Aolserver using telnet , the server waits one of HTTP comands from me. But it would be good to have full access to the socket , from TCL script, without being obliged to send HTTP GET or other requests= . ns_register_proc - unfortunately doesn't give such a possibility, I can h= ave access to the socket but only after HTTP request has been sent. ns_socklistencallback - it gives full controll over the channell , but process only one client at the moment. It will accept connections from several clients, but will process only the first one. The others will blo= ck untill the first one will disconnect. Please , if anyone knows the solution for this, share it with me . -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
[AOLSERVER] Offensive spam
Hey, I assume that other members of the AOLserver community have received similar emails from grou.ps. Who is behind this and how can we put a stop to this spam? I think that it is important for our community to actively reject this type of theft and spam, but how do we do this without confirming valid email accounts? tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] SSL data truncation
Your SF bug report says that you put in a 300 millisecond delay. Where? Even if you think that such a fix is not good, it would be helpful to at least know what works. It might help track down the bug, or help others start looking at something smaller than ns_httpsget/post. You also talk about truncation, but then the truncation stops if the received data goes above 81000. It might be a good idea to narrow down when the bug appears (what byte value) and when it goes away again. This might suggest something. tom jackson On Wed, Jul 15, 2009 at 3:35 PM, John Carusojcar...@arenasolutions.com wrote: On Wednesday 03:17 PM 7/15/2009, Scott Goodwin wrote: Tell me what version of OpenSSL you're running. OpenSSL 0.9.8k. It's been happening for many years with different OpenSSL versions as well. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_adp_parse issue
Francesco,
Okay here is the thing: you keep using two servers to diagnose an API
which doesn't appear in any of the code you are using. Just use one page
on the server that you think does not work. You don't need to include
any other page, parse any other page or do anything fancy. If you do,
you will have no idea where the problem is or what it is. You could keep
guessing for years until you do this, and it is a waste of your time and
everyone else's time.
But even if your three page/two server setup could narrow down the
problem, the fact is that the problem doesn't always happen. If it
doesn't happen every time, the failure must be due to specific
conditions or configurations, or a bug in your code.
Right now you have no idea what is going on, and neither do I, but your
debugging method cannot ever work to find out. My opinion is learn to
use telnet to request a simple page which reproduces the behavior.
My opinion is that it is just a programming bug and has nothing to do
with ns_adp_abort, but you are not going to find the bug by using
ns_httpget, and you may not find it using a regular browser or command
line client like wget of curl.
tom jackson
On Mon, 2009-07-06 at 21:04 +, Francesco Petrarch wrote:
Ok, I got this, this time I really do, and it should be easily
replicated by anyone.
I have this filter:
ns_register_filter preauth GET /*.html decode_url
ns_register_filter preauth POST /*.html decode_url
ns_register_filter preauth HEAD /*.html decode_url
proc decode_url { why } {
set page [ns_adp_parse -file /www/website/template.adp]
ns_return 200 text/html $page
return filter_return
}
template.adp then does a [ns_conn url] etc etc to serve the correct
content into the template. However, if template.adp does a
ns_adp_abort itself (or any of the files which may be ns_adp_parsed or
ns_adp_included etc) then the thread gets corrupted.
Here is a simple way on how to replicate this with just a single
server.
First create a filter:
ns_register_filter preauth GET /*.html decode_url
ns_register_filter preauth POST /*.html decode_url
ns_register_filter preauth HEAD /*.html decode_url
proc decode_url { why } {
set page [ns_adp_parse -file /PATHTOWWWFILES/testpage.html]
ns_return 200 text/html $page
return filter_return
}
Second create the file testpage.adp
%
set ret
set max 200
set count 0
while {$count$max} {
incr count
set page [ns_httpget
http://www.YOURWEBSITE.com/testpage.html?count=$count;]
if {[string length $page]!=0} {
if {$count==50} {
append ret First 50 requests okbr
}
if {$count==$max} {
append ret Request Max okbr
}
} {
append ret request $count had filesize [string length
$page]br
}
}
ns_puts $ret
%
Finally create testpage.html (note the last line Testdata, so we have
some text on the page)
%
set_form_variables 0
if {$count==50} {
# ns_adp_abort
}
%
Testdata
Now, in your webbrowser go to http://www.YOURSITE.com/testpage.adp,
after a few seconds you should see a webpage like
First 50 requests ok
Request Max ok
You can reload this a number of times and always see the same thing.
Now edit testpage.html by removing the comment on the ns_adp_abort
line
Then reload http://www.YOURSITE.com/testpage.adp
You will see something like this:
request 50 had filesize 0
request 113 had filesize 0
request 176 had filesize 0
Request Max ok
Keep reloading for lots of fun!
I have tried modifying the filter to account for a redirect and
returning filter_break or filter_ok but with no luck. What is
interesting to note is that you can put ns_log notices in the filter,
and the requests that return 0 in size do get into the filter, but the
ns_adp_parse just will just return an empty string. I am thinking
that this is because 'ns_adp_abort' is still set for the thread?
I am uncertain if this is a bug. If there something other than
ns_adp_abort I could use after a ns_returnredirect which will return
all the way back up to the filter? I believe ns_adp_return will just
return 1 level up and ns_adp_break gives the same problem of also
breaking future connections. Or is there a way I can ns_adp_unabort
at the start of the filter?
_Peter
__
Date: Mon, 6 Jul 2009 10:42:28 -0700
From: t...@rmadilo.com
Subject: Re: [AOLSERVER] ns_adp_parse issue
To: AOLSERVER@LISTSERV.AOL.COM
Did I miss something?
Your script below does not contain a redirect and does not contain
ns_adp_abort
Re: [AOLSERVER] ns_adp_parse issue
Jeff,
Thanks for that bit of help. As far as I can tell everything seems to
work correctly. I setup a slightly different test to demonstrate the
behavior.
Register filters in the private library:
proc ::test_adp_abort {arg why} {
switch -exact -- $arg {
file {
ns_return 200 text/plain [ns_adp_parse -file test.adp]
}
info {
ns_adp_exception adp_state
ns_return 200 text/plain ns_adp_exception_state = $adp_state
}
}
return filter_return
}
ns_register_filter preauth GET /adp-file ::test_adp_abort file
ns_register_filter preauth GET /adp-info ::test_adp_abort info
Place test.adp in the pageroot:
%
ns_return 200 text/plain hi
ns_adp_abort
ns_log Notice After abort, I shouldn't be running
%
Visit urls in a particular order:
1. use ns_adp_parse -file to load test.adp
http://localhost:8000/adp-file
(browser returns hi)
Error log shows an error:
Error: Tcl exception:
while executing
ns_adp_abort
invoked from within
ns_adp_parse -file test.adp
(file arm line 2)
invoked from within
switch -exact -- $arg {
file {
ns_return 200 text/plain [ns_adp_parse -file test.adp]
}
info {
ns_adp_exception adp_state
ns_...
(procedure ::test_adp_abort line 3)
invoked from within
::test_adp_abort file preauth
---
2. Now check the adp status
http://localhost:8000/adp-info
(browser returns ns_adp_exception_state = abort)
Nothing in the error logs
3. Revisit adp-file
http://localhost:8000/adp-file
(browser returns server error)
Error log has:
Error: Tcl exception:
while executing
ns_adp_parse -file test.adp
(file arm line 2)
invoked from within
switch -exact -- $arg {
file {
ns_return 200 text/plain [ns_adp_parse -file test.adp]
}
info {
ns_adp_exception adp_state
ns_...
(procedure ::test_adp_abort line 3)
invoked from within
::test_adp_abort file preauth
Notice that now ns_adp_parse is returning the error, not the call to
ns_adp_abort.
4. Directly access test.adp:
http://localhost:8000/test.adp
(server returns hi)
5. Revisit adp-info
http://localhost:8000/adp-info
(server returns ns_adp_exception_state = ok)
*** Analysis ***
Working exactly like Jeff said. One thing to notice is that direct
access of the test.adp file resets the state.
Why does this happen? Using ns_adp_abort inside a tcl context, or using
it outside of an adp context. Even ns_adp_parse is outside of the adp
context, there is no toplevel handler.
The basic problem is trying to reuse a directly accessed adp as input to
ns_adp_parse. This doesn't work in general, although you might find some
cases where it does work.
tom jackson
On Tue, 2009-07-07 at 13:08 -0700, Jeff Rogers wrote:
Jeff Rogers wrote:
proc decode_url { why } {
set page [ns_adp_parse -file /www/website/template.adp]
ns_return 200 text/html $page
ns_adp_exception state
ns_log notice adp exception state is $state
catch {ns_adp_return}
return filter_return
}
Whoops, this won't work. The ns_adp_return will let the adp interpreter
run *after* the next eval, but the next time just exits. (an additional
piece of debugging code was hiding this from me initially)
I think a better approach (lacking ns_adp_ctl reset) is
proc decode_url { why } {
set page [ns_adp_parse -file /www/website/template.adp]
ns_return 200 text/html $page
ns_adp_exception state
ns_log notice adp exception state is $state
# reset the adp state
set dn [open /dev/null w]
ns_adp_ctl chan $dn
catch { ns_adp_ctl chan }
close $dn
return filter_return
}
Of course, I had to dig around in the code a bit to figure out how to
call NsAdpReset, but on the plus side the AOLserver code is much more
pleasant to dig around in than many other programs.
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_adp_parse issue
Did I miss something?
Your script below does not contain a redirect and does not contain
ns_adp_abort. Why do you think this script tells you anything about your
problem?
But one thing seems apparent: this adp script should return nothing. It is
just a script, it doesn't produce content. If no content is produced, then
it is working.
On Sun, Jul 5, 2009 at 10:32 PM, Francesco Petrarch
f_petra...@hotmail.comwrote:
Hu, on further testing I guess I was pretty much completely wrong
I guess I spoke too soon.
I have tried to turn on all error logging but have come up with nothing. I
am able to intentionally kill a thread, though I am not entirely sure how :S
There are some pages I can request that will kill an additional ADP thread,
at first I thought this was simply the ns_returnredirect or the ns_adp_abort
but seems to be something else since I can't duplicate it with all
redirects/aborts (although it does happen each time these specific pages are
requested). For example, the profile.html page redirects to the
login/signup page if you are not logged in and if it does so causes the
problem
Here is the latest.
I have a script at http://SERVERA/test .adp
%
set count 0
while {$count2} {
incr count
set page [ns_httpget http://SERVERB/file.html?count=$count;]
ns_log notice page $count size [string length $page]
if {[string length $page]2} { break }
}
%
it basically requests a file 2 times from SERVERB (which also hosts the
aforementioned profile.html page) and stops if the file size is small. The
file file.html contains just a single word test. If .html is not in
ns_section ns/server/serverb/adp then the problem does not happen, it
only occurs when file.html may or may not need the parser. In the event
that .html is not in the ns_section and is static, fastpath settings do not
seem to make a difference, everything works.
Now from my observations, it appears that when a file such as the
aforementioned profile.html page is requested and the redirect happens, that
thread becomes contaminated. The next time that contaminated thread is used
for an adp the problem appears. It appears as though the ns_adp_abort
value is still set and adps can not be parsed, even if the file.html file
has no % % tags, if .html is set as a file to be parsed then an empty
string is returned, however, just once. Once this thread is closed again it
seems fixed for future requests. However, this contamination is only
cleaned when the thread tries to serve/parse an adp, if it's serving static
content the contamination is not removed. Going back to the filters I had
originally I believe the request still comes in for the page with the adp,
the filter can log info, but ns_adp_parse just returns an empty string.
Anyways, I'll be digging further into this, trying to see what exactly is
contaminating the thread as it appears now. To repeat, this problem is
repeatable everytime profile.html is requested, and if it is related to
ns_adp_abort or ns_adp_redirect there must be other variables to the problem
since I can not seem to duplicate the problem with just those functions.
Also I do not use ns_perms, I simply set a cookie once someone is logged
in.
I don't suppose there is a ns_adp_unabort
_Peter
--
Date: Sun, 5 Jul 2009 18:50:53 -0700
From: t...@rmadilo.com
Subject: Re: [AOLSERVER] ns_adp_parse issue
To: AOLSERVER@LISTSERV.AOL.COM
On Sun, Jul 5, 2009 at 6:13 PM, Francesco Petrarch f_petra...@hotmail.com
wrote:
It's been awhile since I have looked into this and I think I have tracked
it down. In short, I now believe that ns_adp_abort aborts all current
threads/adps, not just the one currently executing the ns_adp_abort line.
Therefore, when these blank screens were appearing on my site to VISITOR A
it was because another visitor (VISITOR B) had visited a page which called
ns_adp_abort (for example after a ns_returnredirect) before the VISITOR A
thread finished.
Does that make sense? Can someone with more experience in the code take a
look and let me know?
No, makes no sense. This is essentially impossible. Each thread executes
pretty much independently of all others. The problem probably has to do with
the adp configuration...error without signaling an error, which is a
configuration which is necessary to keep complex sites up if one component
is not working.
tom jackson
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
Windows Live helps you keep up with all your friends, in one
place.http://go.microsoft.com/?linkid=9660824
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists
Re: [AOLSERVER] ns_adp_parse issue
On Sun, Jul 5, 2009 at 6:13 PM, Francesco Petrarch f_petra...@hotmail.comwrote: It's been awhile since I have looked into this and I think I have tracked it down. In short, I now believe that ns_adp_abort aborts all current threads/adps, not just the one currently executing the ns_adp_abort line. Therefore, when these blank screens were appearing on my site to VISITOR A it was because another visitor (VISITOR B) had visited a page which called ns_adp_abort (for example after a ns_returnredirect) before the VISITOR A thread finished. Does that make sense? Can someone with more experience in the code take a look and let me know? No, makes no sense. This is essentially impossible. Each thread executes pretty much independently of all others. The problem probably has to do with the adp configuration...error without signaling an error, which is a configuration which is necessary to keep complex sites up if one component is not working. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_db exec retrieve stdout (postgresql copy to)
Matt, This is probably a mistake. The COPY command creates a text file (or CLOB), which _must_ be parsed if you want to create records and fields. Just use dml to get the data into an ns_set, or the C API equivalent. All CSV handlers have problems. PostgreSQL's COPY seems to have difficulty with quoting, meaning the quote character might appear unescaped in a field, causing errors. ns_csv* has problems with record boundaries: each record must be on one line, so \n, etc. don't pass through. In general, it is a bad idea to convert your data to a more ambiguous format and then try to recover it unchanged. That is why we have ns_set, arrays and lists. I work with pg's COPY because it is fast, extremely fast, but your data must be perfect to use it. You could try using COPY to a file then see how fast ns_csvget is, but I assume that the regular database API will be faster, and over time, using a database API will be much less prone to errors. If this isn't important, you might reconsider using a database in the first place. tom jackson On Tue, Jun 9, 2009 at 8:49 AM, Matt Haggard haggar...@gmail.com wrote: PostgreSQL has a COPY TO command ( http://www.postgresql.org/docs/8.1/static/sql-copy.html ), which outputs query results in CSV format. It would be neat if I could execute the command using ns_db dml rather than parsing in TCL. COPY TO writes the result to STDOUT (or a file, but I'd like to not use a file). Any help? I've been going through the ns_db source, but I dead-end at NsDbGetDriver(); Thanks, Matt Haggard -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_adp_parse issue
I would try testing on a page from a local server so you can verify that
the server is returning what you expect.
Also, log the actual content when it is not what you expect. you have
access to the $page value, maybe it is an error message.
Obviously something is failing, but nothing you have shown us narrows
the location of the possible problem. Is it the adp, the configuration
of adps or the error log, or is the foreign server screwing up, or your
network. The fact that the code works most of the time means you
probably need to look at every step and the value of every step.
That said, my guess something is going wrong with the ns_httpget
operation, maybe it gets oveloaded, times out or something else.
tom jackson
On Fri, 2009-06-05 at 20:56 +, Francesco Petrarch wrote:
Thanks for the suggestions, however they have not provided any
positive results.
ns_adp_ctl stricterror 1 - did not yield any additional errors/notices
(nothing is written to the log files when this happens)
ns_param StackSize 2048000 - only succeeding in causing aolserver to
crash with could not allocate memory' errors after a day of running
For interest sake, the way I test this is from a different server I
have this little script:
%
set count 0
while {$count1000} {
incr count
set page [ns_httpget http://www.mysite.com/members.html]
ns_log notice page $count size [string length $page]
if {[string length $page]1000} { break }
}
%
And some sample lines form that logfile are:
[05/Jun/2009:16:26:47][31912.2950486928][-default:394-] Notice: page
86 size 38660
[05/Jun/2009:16:26:47][31912.2950486928][-default:394-] Notice: page
87 size 38660
[05/Jun/2009:16:26:47][31912.2950486928][-default:394-] Notice: page
88 size 38660
[05/Jun/2009:16:26:48][31912.2950486928][-default:394-] Notice: page
89 size 38660
[05/Jun/2009:16:26:48][31912.2950486928][-default:394-] Notice: page
90 size 38660
[05/Jun/2009:16:26:48][31912.2950486928][-default:394-] Notice: page
91 size 38660
[05/Jun/2009:16:26:48][31912.2950486928][-default:394-] Notice: page
92 size 38660
[05/Jun/2009:16:26:48][31912.2950486928][-default:394-] Notice: page
93 size 38660
[05/Jun/2009:16:26:48][31912.2950486928][-default:394-] Notice: page
94 size 87
When I didn't return the javascript on a failure the size would be 0
not 87.
I have yet to try Dossy's suggestion of using the startpage because I
would rather try and find out what the issue actually is incase there
are other issues involved with it. Plus since the javascript is
currently a working bandaid fix I am in no hurry, the only real
downside with my hack is a search engine may index it.
What puzzels me is that, as I mentioned, these 0 sized ns_adp_parse
calls seem to happen at random, when the page it happens on is
reloaded things work fine, so all the input data is the same.
Combined with the fact when I attempt to call ns_adp_parse multiple
times when it does return a 0 sized string, all successive calls also
return a 0 sized string leads me to think something in the
ns_adp_parse functionality is failing without throwing an error...
maybe a lock contention somewhere? I even put a ns_sleep 1 between
ns_adp_parse calls but even after 10 seconds it returns a 0 length
string.
I will now be digging into the Aolserver code itself to see what I can
find logging to the log file between every line of code if I must
to find out where this is failing.
Thanks for the help so far,
_Peter
Date: Fri, 29 May 2009 09:06:11 -0700
From: t...@rmadilo.com
Subject: Re: [AOLSERVER] ns_adp_parse issue
To: AOLSERVER@LISTSERV.AOL.COM
Maybe there is an error which is silently caught in ns_adp_parse.
Try this before sourcing your adp:
ns_adp_ctl stricterror 1
tom jackson
On Fri, 2009-05-29 at 14:48 +, Francesco Petrarch wrote:
I appear to be having a problem with ns_adp_parse and ns_adp_eval.
I
currently run the latest code 4.5.1, upgraded from 3.13 with Jerry
Asher's patches for unix sockets for multiple hosts. Everything
worked fine on 3.13 but now I have intermittent issues with 4.5.1
(I
had acutally upgraded in November and have the issue with all 4.5
updates since then).
My website serves about 140 million requests a month, about 3
million
of those are .html requests.
I don't use ACS, everything is custom coded and my CMS is based on
the
following filter:
ns_register_filter preauth GET /*.html decode_url
ns_register_filter preauth POST /*.html decode_url
proc decode_url { why } {
ns_return 200 text/html [ns_adp_parse
-file /www/website/template.adp]
return filter_return
}
Essentially, every .html page goes through this filter, and the
file /www/website/template.adp is a graphical template/layout for
the
site and dynamically includes the content for the give [ns_conn
url].
As I mentioned this worked
Re: [AOLSERVER] AOLServer documentation
I have copies of some docs here: http://rmadilo.com/files/as23docs/index.html http://rmadilo.com/files/docs/toc.html http://rmadilo.com/files/nsapi/ And various other info under: http://rmadilo.com/files/ and Tcl, AOLserver and related docs: http://junom.com/document/ You can search the almost the entire AOLserver source code at: http://junom.com/gitweb/gitweb.perl?p=aolserver.git (using pickaxe) Also the changelogs might help in the last one. tom jackson On Wed, 2009-06-03 at 11:16 +0200, Gahan, Mike wrote: Hi all, Where is the AOLserver online doc these days? I was using http://dev.aolserver.com/wiki/Tcl_API but that is erroring now. Mike Gahan Architect / Analyst / Developer / Fixer AOL (EU) Limited Tel: +44 (0) 20 7348 8857 Email: mike.ga...@corp.aol.com AIM: mikegahanuk 68 Hammersmith Road London W14 8YW www.aol.co.uk www.aol.co.uk/mediaspace Try our new products http://beta.aol.co.uk/ This e-mail, its content and any files transmitted with it are confidential and may be legally privileged. If you receive it in error or are not the intended recipient you must not copy, distribute or take any action in reliance upon it. Instead, please notify us immediately by telephoning +44 (20) 7348 8000 - ask for the Legal Department - and delete the material from your systems. In the UK, AOL Europe is a business carried on by AOL (UK) Limited, a company registered with the Registrar of Companies for England and Wales under number 3462696. Registered office: 68 Hammersmith Road, London, W14 8YW. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Is ns_info threads safe to use?
This is an interesting topic, but I can't think of anything to be gained from a list of threads. Who cares that a thread exists? But if you are just concerned with threads, you can use ns_pools. All worker threads in AOLserver are in some named thread pool. If you don't use threadpools, all requests, no matter how many virtual servers you use are handled by the default threadpool. A query using ns_pools can give you a current thread count. Also you should know that every ns_log (error log) line has information about the threadpool and the thread id and every error log has the process id dot thread id just after the timestamp. If you wrote a script to examine the last part of the log file, you could discover which threads were active. Personally I would abandon the use of a list of living threads as a measure of anything. When AOLserver goes dark threads usually don't go away. tom jackson On Tue, 2009-06-02 at 12:12 +0200, Gustaf Neumann wrote: It certainly depends on what your application needs. There is no principal problem obtaining the thread id (eg. ns_thread id, ::thread::id). One could either use the sketched approach and simply record whatever the application needs, or ffone can to extend the xotcl-request monitor to track this information as well. just to get the information about running connection threads from the xotcl-request-monitor, use throttle running. best regards -gustaf neumann Sep Ng schrieb: Hi Gustav! Thanks for the info. I'm afraid xotcl-request-monitor may not be good enough if I do not have the thread ids, although I guess I could rewrite it to work with what xotcl-request-monitor provides. I did not know of one such monitoring page on OpenACS. I will take a look and see what I can find. On Jun 2, 3:42 pm, Gustaf Neumann neum...@wu.ac.at wrote: Sep Ng schrieb: Is there a way in AOLserver to do live monitoring on threads? We're sort of hoping to get info on what thread ids are running as of the moment on aolserver. The xotcl-request-monitor watches running requests, essentially via defining filters/traces for requsts and using a monitor thread for keeping track of the starts and ends of requests. If there are starts recorded without ends, it knows these requests are still running in some threads. This approach does not depened on ns_info, we use it on all our production sites. Originally i had one version for pure aolserver and one for OpenACS; since a while i just work on the OpenACS version (which is available via the public cvs repository of OpenACS). best regards -gustaf neumann -- AOLserver -http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_adp_parse issue
Maybe there is an error which is silently caught in ns_adp_parse.
Try this before sourcing your adp:
ns_adp_ctl stricterror 1
tom jackson
On Fri, 2009-05-29 at 14:48 +, Francesco Petrarch wrote:
I appear to be having a problem with ns_adp_parse and ns_adp_eval. I
currently run the latest code 4.5.1, upgraded from 3.13 with Jerry
Asher's patches for unix sockets for multiple hosts. Everything
worked fine on 3.13 but now I have intermittent issues with 4.5.1 (I
had acutally upgraded in November and have the issue with all 4.5
updates since then).
My website serves about 140 million requests a month, about 3 million
of those are .html requests.
I don't use ACS, everything is custom coded and my CMS is based on the
following filter:
ns_register_filter preauth GET /*.html decode_url
ns_register_filter preauth POST /*.html decode_url
proc decode_url { why } {
ns_return 200 text/html [ns_adp_parse
-file /www/website/template.adp]
return filter_return
}
Essentially, every .html page goes through this filter, and the
file /www/website/template.adp is a graphical template/layout for the
site and dynamically includes the content for the give [ns_conn url].
As I mentioned this worked fine for 3.13
However, seemly at random times (for a random connection),
[ns_adp_parse -file /www/website/template.adp] will return an empty
string. I have modified the filter to reattempt the ns_adp_parse if
the returned value was an empty string (ie: [string length $page]==0)
and logged the progress, but on a connection where ns_adp_parse
returns the empty string it appears to always return an empty string.
ns_adp_eval also returns an empty string when this happens. With the
above filter the visitor to the website would see just a blank screen
because of course the content was just . To be exact here, I say
empty string, to be more accurate it returns $page where [string
length $page]==0
My current little fix is this:
ns_register_filter preauth GET /*.html decode_url
ns_register_filter preauth POST /*.html decode_url
proc decode_url { why } {
set page [ns_adp_parse -file /www/website/template.adp]
set size [string length $page]
if {$size==0} {
ns_log notice Page [ns_conn url] ns_adp_parse $size in size for
visitor from [ns_conn peeraddr]
set page body onload='window.location.reload()'
}
ns_return 200 text/html $page
return filter_return
}
As you can see, not so much a fix as just creating a little javascript
to reload the page. At least the visitor to the website is no longer
presented with a blank page, but not as ideal as an actual fix to the
empty string issue.
I have thought that this may be an issue with threads or some limit
with handles to databases or maybe the adp parser (I really don't know
how that works). However, given that this issue was not present on
3.13, and no errors are generated I am at a loss. I have also tried
messing with the config, changing just about everything to see if it
made any difference, it didn't.
I will be continuing to try and debug this to confirm
that /www/website/template.adp is being parsed and so on, but before I
got too involved I figured I'd try this list to see if anyone knows of
similar issues.
Thanks for your help,
_Peter
__
Create a cool, new character for your Windows Live™ Messenger. Check
it out
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Question on two AOLserver tickets
One thing which would be nice is to put the Tcl API into a module which as to be loaded, and keep the C API available in the core. It is obviously more work, but the admin has to make a choice to load the Tcl API, which is more open to misuse. I did something like this for a few additional ns_conn options. Gustaf's idea is easier to pull off at least initially, but it shouldn't be tied to an unrelated option like debugging. I leave on debugging for low traffic servers and I can also turn it on/off on a per-namespace basis. tom jackson On Thu, 2009-05-14 at 15:48 -0700, Sep Ng wrote: How about having the proc enable only if debug settings are turned on on AOLserver? -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Question on two AOLserver tickets
Maybe calling the API should result in a ns_log Warning to indicate a
potential crash.
tom jackson
On Thu, 2009-05-14 at 13:26 -0700, Jade Rubick wrote:
I'm just happy we figured it out.
We were using this call:
set connections [ns_server active]
But it wasn't in a scheduled proc, so I just moved it behind a
password protection section, and put a warning around it. We seldom
(never) used that page anyway. I think a bot may have found it or
something.
Jade
Jade Rubick
Director of Development
TRUiST
120 Wall Street, 4th Floor
New York, NY 10005 USA
jrub...@truist.com
+1 503 285 4963
+1 707 671 1333 fax
www.truist.com
The information contained in this email/document is confidential and
may be legally privileged. Access to this email/document by anyone
other than the intended recipient(s) is unauthorized. If you are not
an intended recipient, any disclosure, copying, distribution, or any
action taken or omitted to be taken in reliance to it, is prohibited.
On May 14, 2009, at 12:33 PM, Jim Davidson wrote:
Yup -- should really have been documented better -- sorry about
that.
Anyway, what is the monitoring attempting to dig up? There may some
other safe ways to get the same.
-Jim
On May 14, 2009, at 2:04 PM, Jade Rubick wrote:
Ironically, we have some monitoring code that does use that
functionality.
So our monitoring is killing our servers. Nice!
I'm removing that code now.
Jade Rubick
Director of Development
TRUiST
120 Wall Street, 4th Floor
New York, NY USA
jrub...@truist.com
+1 503 285 4963
+1 707 671 1333 fax
www.truist.com
The information contained in this email/document is confidential
and may be legally privileged. Access to this mail/document by
anyone other than the intended recipient(s) is unauthorized. If
you are not an intended recipient, any disclosure, copying,
distribution, or any action taken or omitted to be taken in
reliance to it, is prohibited.
On Thu, May 14, 2009 at 10:19 AM, Jim Davidson
jgdavid...@mac.com wrote:
Hi,
Do you have some sort of background job that calls
ns_server active (or similar) regularly? That could
lead to random crashes. The description in
http://dev.aolserver.com/trac/ticket/152 is accurate: The
code, by design, is not strictly safe as it's assumed to
only be used interactively and occasionally as part of
debugging and performance monitoring/optimization.
To make it safe would require adding mutex locks around
areas that are assumed read-only and/or single-threaded
which could possibly lead to lock contention. I can't say
it those assumptions have ever been proven true or false
but that was my thinking when the code was first written.
-Jim
On May 14, 2009, at 4:16 AM, Sep Ng wrote:
Hi,
I'm trying to debug an AOLserver crash and the
point of crash seems to
be AppendConn in NS_GetProcInfo... I will post the
stack trace after
just for reference.
Looking through the ticket tracker on AOLserver, I
found two tickets
of particular interest:
http://dev.aolserver.com/trac/ticket/325
-- My question with this ticket is was it ever
resolved?
and the second ticket:
http://dev.aolserver.com/trac/ticket/152
-- This problem should only happen if the command
ns_server was
called in a multi-threaded environment, right?
Here is the call stack trace I'm working with.
I'm more interested in
Ticket #325 as it may be related to my problem.
- Call Stack Trace -
calling call entry
argument values in
hex
location type point
(? means dubious
value)
kpedbg_dmp_stack()+ call B5B81884
B45FFB74 ? 0 ?
219
kpeDbgCrash()+72 call B5B75E14
0 ? 5 ? 0 ?
80BD810
Re: [AOLSERVER] Question on two AOLserver tickets
On Thu, 2009-05-14 at 18:08 -0400, Jim Davidson wrote: Good idea. Maybe it would make sense to disable it by default with some config flag to enable? I was thinking the same, but I wasn't sure how many people actually use this command. This must be one of a very few commands that I have never used, so I'm far from informed on what it is used for. Sent from my iPhone Sent via a local network shared with an iTouch. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Diagnosing an AOLserver performance problem
On Wed, 2009-05-13 at 09:54 -0700, Janine Sisk wrote: Are you referring to Rob Mayoff's threadpool code from 2004? Does it still work in 4.5.1? No AOLserver has url based, connection threadpools built in. But by default they don't get configured very well. Rob's code was more like a worker thread pool, something like warmed up, non-connection threads. There isn't any need to modify any code, just configure them. Use nscp and telnet in and check out the ns_pools command. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] ns_httptime format error in AOL 4.5.1
Looking at the source for a strftime, I see only two possible problems:
1. The format passed to sprintf is wrong, or
2. the case 'd' branch could be missing the continue, so that case 'e'
runs, and replaces the result of 'd':
case 'd':
pt = _conv(t-tm_mday, %02d, pt, ptlim);
continue;
case 'e':
pt = _conv(t-tm_mday, %2d, pt, ptlim);
continue;
Note that _conv is simply:
static char *_conv(const int n, const char *format, char *pt, const char *ptlim)
{
char buf[32];
sprintf(buf, format, n);
return _add(buf, pt, ptlim);
}
Since other formats rely on %d, you might also look at the results of
formatting %x, %D and %F. But you would have to edit httptime.c with a
new format string:
strftime(buf, 40, %x, tmPtr);
tom jackson
On Tue, 2009-05-12 at 18:33 +0400, Alexey Pechnikov wrote:
Hello!
On Tuesday 12 May 2009 17:29:38 Gustaf Neumann wrote:
Mr Spock would say: fascinating.
We know at least, that the environment is indeed
not the problem. Can you check, whether both programs
use the same strftime function?
such as:
$ nm time-format| fgrep strftime
U strftime@@GLIBC_2.2.5
$ nm /usr/local/src/aolserver-4.5.1/nsd/libnsd.so | fgrep strf
U strftime@@GLIBC_2.2.5
I didn't build AOL with debugging symbols.
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] ns_httptime format error in AOL 4.5.1 [solved]
Great! I was actually thinking that it would be much easier in this
particular case to avoid strftime, since all of the variability in the
format is removed. It is huge overkill, and obviously broken in
non-english languages.
tom jackson
On Tue, 2009-05-12 at 20:36 +0400, Alexey Pechnikov wrote:
Hello!
I'm sorry but the problem is produced by patch for non-english locale.
This patch use wrong format string:
snprintf(buf, 40, %s, %d %s %d %02d:%02d:%02d GMT,
I did fix the bug and I'm think this patch may be added to upstream.
Patch is here:
http://mobigroup.ru/files/aol4.5.1/httptime.c.diff
Thanks for help!
=
--- httptime.c.orig 2003-01-18 22:24:20.0 +0300
+++ httptime.c 2009-05-12 20:14:24.0 +0400
@@ -27,6 +27,9 @@
* version of this file under either the License or the GPL.
*/
+static char *weekdays_names[7] =
+{ Sun, Mon, Tue, Wed, Thu, Fri, Sat };
+
/*
* time.c --
@@ -92,11 +95,14 @@
}
/*
- * This will most likely break if the locale is not an english one.
+ * Using snprintf instead of strftime to always use english names
* The format is RFC 1123: Sun, 06 Nov 1997 09:12:45 GMT
*/
-strftime(buf, 40, %a, %d %b %Y %H:%M:%S GMT, tmPtr);
+snprintf(buf, 40, %s, %02d %s %d %02d:%02d:%02d GMT,
+weekdays_names[tmPtr-tm_wday], tmPtr-tm_mday,
+month_names[tmPtr-tm_mon], tmPtr-tm_year + 1900,
+tmPtr-tm_hour, tmPtr-tm_min, tmPtr-tm_sec);
Ns_DStringAppend(pds, buf);
return pds-string;
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Diagnosing an AOLserver performance problem
On Tue, 2009-05-12 at 22:23 +0200, Gustaf Neumann wrote: Janine Sisk schrieb: When I first started working on this problem, the settings were ns_param maxconnections 5 ns_param maxthreads 5 ns_param minthreads 5 maxconnections of 5 is for most applications to small. The term maxconnections is misleading since it does not mean maximum number of incoming connections, but is a counter per thread, when it will shut down. With the value above, every thread will exit and be recreated after 5 requests. If the application is well written, no garbage should be left after the request, therefore the value can be set to e.g. 1000. This sounds like great general advice. If I had to boil down a few rules, I would say at bare minimum: maxconnections 5 * maxthreads. But 10 to 20 times would be much better. Also, unless you have a memory leaking application, start with a minimum of 100 for maxconnections. You should also be using threadpools. If you don't, then every request gets dumped into a default threadpool that is shared by all virtual servers. What this means is that it is possible for multiple interps to exist in each thread, up to one for each virtual server. At a bare minimum, each virtual server should have its own default threadpool which matches every request to that vs. So the problem with one nsd-process-wide threadpool is the memory footprint of each thread will be at least the sum of each vs interp. Maybe I should point out that one advantage of threads is to reduce the memory footprint by specializing the interp code. If you setup a number of threadpools, you can partition your requests into static and dynamic (at the very least). A few threads can handle many static requests. You should also partition your requests by need run tcl code/filters. Do you want to require filter processing for css and basic image files? Personally I would use a different port for pure static content. I have used publicfile: http://cr.yp.to/publicfile.html but I have heard that AOLserver is pretty fast at static file handling (I'm trying to catch up). tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] ns_httptime format error in AOL 4.5.1 [solved]
On Tue, 2009-05-12 at 20:36 +0400, Alexey Pechnikov wrote: Hello! I'm sorry but the problem is produced by patch for non-english locale. This patch use wrong format string: snprintf(buf, 40, %s, %d %s %d %02d:%02d:%02d GMT, I did fix the bug and I'm think this patch may be added to upstream. At first I thought that you were fixing a bad patch. But the diff isn't a fix of the bug, it is a good replacement for stupidly-generic code (which we still are using in AOLserver). We are still left with the question of how the original problem was created. From my research it appears that the locale would never affect the numbers (it couldn't be wrong with the day format and right with the time format), so we could change the locale with no effect on the output. We have evidence that the locale had no effect. My best guess is that someone created an almost perfect patch to the code you were using. It fixed the day-name and month-name problems, but it screwed up the day-number format. But your patch does not record the bug, just the original code, which would not produce the output we have been using. What gives? You include the buggy snprintf line above, but this isn't in the patch. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] ns_httptime format error in AOL 4.5.1
I also looked into this, I think Gustaf must be right. The reason is that the AOLserver C code passes in the format string, which is clearly correct. %d is the zero padded day number [01-31]. After the call, the modified buffer is sent back as-is, so the bug isn't in AOLserver. The good news, if there is any here is that to set the time formats, you only need to adjust the LC_TIME database, using localedef or locale-gen. Here is a somewhat helpful link: http://www.regatta.cs.msu.su/doc/usr/share/man/info/ru_RU/a_doc_lib/files/aixfiles/LC_TIME.htm Aside from this issue, the code has obvious limitations built into the design, at least ns_parsehttptime. For some reason, ns_parsehttptime uses a return value of 0 to indicate failure. Probably a value of NULL should have been used instead. However, this will only be a problem if we need to maintain documents created in 1970. The other issue is that ns_parsehttptime, which is based upon RFC 1123, should be somewhat forgiving in parsing the time components. It clearly fails when the day number is 1-9 with a leading blank space. Probably the MakeNum API helper should be modified to test the result ranges for each number, and somehow provide feedback on how much buffer was consumed. tom jackson On Mon, 2009-05-11 at 09:02 +0200, Gustaf Neumann wrote: Dear Alexey, From the output, i start to believe that the setting of LANG/LOCALE is not the problem, but i am getting the suspicion that there is a bug in the c-library of your operating system. What operating system do you use? Rationale: aolserver uses strftime(..., ..., ... %d ..., ...) to format the day of month. Since strftime() is defined in the C-library, the bug should be there. -gustaf neumann Alexey Pechnikov schrieb: Hello! On Sunday 10 May 2009 23:42:08 Tom Jackson wrote: I wonder if you have to set this before your start nstclsh? $ LANG=en_US.UTF-8 nstclsh % ns_httptime 0 Thu, 1 Jan 1970 00:00:00 GMT You might also see what my replacement proc produces, or just try in nstclsh: $ nstclsh % set seconds 0 0 % clock format $seconds -format %a, %d %b %Y %H:%M:%S %Z -gmt 3 Thu, 01 Jan 1970 00:00:00 GMT But I just noticed something strange: % ns_parsehttptime [ns_httptime 0] invalid time: Thu, 01 Jan 1970 00:00:00 GMT % ns_parsehttptime Thu, 01 Jan 1970 00:00:00 GMT invalid time: Thu, 01 Jan 1970 00:00:00 GMT % ns_parsehttptime Thu, 1 Jan 1970 00:00:00 GMT invalid time: Thu, 1 JAn 1970 00:00:00 GMT On the last one, notice that the month gets converted to JAn. Hm, I'm get the same result for my ru_RU.UTF-8 locale. Best regards, Alexey Pechnikov. http://pechnikov.tel/ -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Fwd: [AOLSERVER] ns_httptime format error in AOL 4.5.1
On Sun, 2009-05-10 at 22:03 +0400, Alexey Pechnikov wrote: Hello! On Sunday 10 May 2009 21:15:03 Tom Jackson wrote: Alexey: you can use the nstclsh shell to test if changing the locale makes any difference (in same location as your nsd file): ./bin/nstclsh % set ::env(LANG) en_US.UTF-8 % ns_httptime 0 Thu, 01 Jan 1970 00:00:00 GMT % I'm get wrong result: $ nstclsh % set ::env(LANG) en_US.UTF-8 en_US.UTF-8 % ns_httptime 0 Thu, 1 Jan 1970 00:00:00 GMT % I wonder if you have to set this before your start nstclsh? Anyway, whatever is causing the problem has to do with your setup. At least you can test it with nstclsh and once solved there should work for your AOLserver install. You might also see what my replacement proc produces, or just try in nstclsh: % set seconds 0 0 % clock format $seconds -format %a, %d %b %Y %H:%M:%S %Z -gmt 1 If that works/doesn't you might also try whatever tclsh came with your system. But I just noticed something strange: % ns_parsehttptime [ns_httptime 0] invalid time: Thu, 01 Jan 1970 00:00:00 GMT % ns_parsehttptime Thu, 01 Jan 1970 00:00:00 GMT invalid time: Thu, 01 Jan 1970 00:00:00 GMT % ns_parsehttptime Thu, 1 Jan 1970 00:00:00 GMT invalid time: Thu, 1 JAn 1970 00:00:00 GMT On the last one, notice that the month gets converted to JAn. Strange. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_httptime format error in AOL 4.5.1
On Sat, 2009-05-09 at 13:18 +0400, Alexey Pechnikov wrote:
Hello!
As I did find
[ns_httptime 0]
Thu, 1 Jan 1970 00:00:00 GMT
It's wrong! Correct value is Thu, 01 Jan 1970 00:00:00 GMT. So
cache control by headers If-Modified-Since and If-None-Match
refused.
You can use this as a temporary replacement:
proc ::ns_httptime { seconds } {
return [clock format $seconds -format %a, %d %b %Y %H:%M:%S %Z -gmt 1]
}
tom jackson
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] ns_httptime format error in AOL 4.5.1
On Sat, 2009-05-09 at 13:36 -0700, Jeff Hobbs wrote: On 09/05/2009 1:32 PM, Tom Jackson wrote: Anyone know if the Tcl clock functions are thread-safe? Yes, they are. 8.4 had a few issues earlier on, but 8.5 has a completely rewritten 'clock' that certainly does away with those issues. How about the latest 8.4 version? Nothing wrong with 8.5, but there are a few incompatibilities between the two. This may not be important, my only reason for asking is the potential problem with a non-english locale, but that seems just as likely to affect Tcl. The code comment may just affect the month/day names. I think Alexey's issue must be coming from some other code path. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] ns_httptime format error in AOL 4.5.1
On Sat, 2009-05-09 at 19:30 -0400, Dossy Shiobara wrote: On 5/9/09 5:14 PM, Jeff Hobbs wrote: The last known OS thread-safety call issues in 8.4 were with gethostbyname, et al and corrected for 8.4.14 (Y2006). The checks for calls like localtime_r have been in since longer. Assuming strftime is mt-safe, all is ok (the man page is silent on that, but it does print into a passed in buffer). I haven't looked at Tcl 8.5 source, but has clock ... -gmt been fixed so that it doesn't diddle with env(TZ) any more, which isn't thread-safe. I remember that there's Tcl mutexes around the env(TZ) diddling, but that isn't safe when an application embeds Tcl and modifies env(TZ) as well but doesn't have access to the mutex that Tcl's clock is using. I was hoping that newclock addressed this, and newclock is in Tcl 8.5, right? Of course this is why we have abstraction APIs, which are so prominent in AOLserver. Even if an API is, at an initial point, exactly the same as the basis API, it is possible, even likely, that changes will be made. If an abstraction API had been introduced early on, then client code would remain protected from expected changes. It is hard to explain the importance of creating a layered API, until you are faced with the need to change an implementation without changing the client API. AOLserver is a very good example of planning for future change. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] auto-redirects url problem
I'll point you to an example, and include the basics here, but the idea
is you need one procedure which will do the mapping, which replaces or
enhances [ns_conn location]. The map is just from your internal virtual
host to the external method:/host. Unfortunately my example AOLserver
was both front end proxy and back end virtual server so the mapping was
pretty easy, but here goes:
The code below is from the bottom of the page:
http://www.rmadilo.com/files/vat/vat.txt
I used [rename] so that no code had to be modified anywhere else. This
is old code, so please forgive the style. You can use a global var, but
you might choose a namespaced variable instead. The global var
Template is setup using a filter. The filter runs for each
connection...ahead of any other filter, so it needs to be registered in
your $private_tcl_library/init.tcl file, or at least before anything
else that needs to use the variable. Basically, your mapping procedure
will run for each connection, or for each scheduled proc that needs the
external address: (also can't figure out why I renamed the old versions
new):
# following is used for getting the correct redirect
rename ns_conn ns_conn_new
proc ns_conn { args } {
if {[string match [lindex $args 0] location]} {
global Template
if {[info exists Template(host)]} {
return http://${Template(host)}
} else {
return [ns_conn_new location]
}
} else {
return [eval ns_conn_new $args]
}
}
rename ns_returnredirect ns_returnredirect_new
proc ns_returnredirect { arg } {
ns_log Debug arg is $arg
ns_log Debug ns_conn url is [ns_conn url]
if {[string match -nocase http* $arg]} {
ns_returnredirect_new $arg
} elseif {[string match -nocase /* $arg]} {
global Template
ns_log Debug Redirecting to
${Template(protocol)}://${Template(host)}$arg
ns_returnredirect_new ${Template(protocol)}://${Template(host)}$arg
} else {
global Template
set full_url [ns_normalizepath [file dirname [ns_conn url]]/${arg}]
ns_log Debug full_url is $full_url
ns_log Debug Redirecting to
${Template(protocol)}://${Template(host)}$arg
ns_returnredirect_new ${Template(protocol)}://${Template(host)}$full_url
}
return
}
InitTemplates segway
# Following procs are for use with OpenACS (VirtualACS)
# You can comment these out if you do not have multiple
# ACS installations running from the same AOLserver.
rename ns_info ns_info_old
proc server_name_for_connection { } {
global Template
if {[info exists Template(server)]} {
return $Template(server)
} else {
# Not in connection Thread
return [nsv_get TemplateParameters defaultserver]
}
}
proc ns_info { option } {
switch $option {
server {
return [server_name_for_connection]
}
default {
return [ns_info_old $option]
}
}
}
More info at in the vat directory: http://www.rmadilo.com/files/vat/
tom jackson
On Mon, 2009-05-04 at 01:28 +0400, Alexey Pechnikov wrote:
Hello!
On Sunday 03 May 2009 21:36:52 Tom Jackson wrote:
If your solution below is working, it seems pretty simple to me, a tiny
wrapper around ns_returnredirect.
Well, can I replace internal ns_returnredirect function to my wrapper with
same name for all interps? AOL 4.5.1 has a lot of features but there are a
few
examples. May I use init script for all interps for this? Is it correct way?
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] auto-redirects url problem
Alexey,
Looks great, you might also look at [ns_conn location] If any internal
Tcl code uses it to build url, they will stop working behind a proxy.
Just one instance: sometimes it is useful to return basic style based
images via http, so image urls might need to be rewritten during an
https session.
tom jackson
On Mon, 2009-05-04 at 12:43 +0400, Alexey Pechnikov wrote:
Hello!
I did write the wrapper for AOL 4.5:
ns_ictl oncreate {
if {[info commands _ns_returnredirect] eq {}} {
rename ns_returnredirect _ns_returnredirect
proc ns_returnredirect {url} {
# reverse-proxy must add header X-Forwarded-Proto: https
# for SSL mode
# HAProxy: reqadd X-Forwarded-Proto:\ https
# Pound: AddHeader X-Forwarded-Proto: https
if {[string equal [ns_set get [ns_conn headers]
X-Forwarded-Proto] https]} {
set proto https
} else {
set proto http
}
_ns_returnredirect $proto://[ns_set get [ns_conn headers]
Host]$url
}
}
}
This work fine for me.
Best regards, Alexey Pechnikov.
http://pechnikov.tel/
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] Tuning Question
I would guess there are a number of problems. First is that the number of threads is way too high compared to the conns per thread. you have a 5:1 ratio, I wouldn't go with anything less than 1:1, and 1:10 would be even better. So 10 threads, 100 conns per thread before exit. Another problem you can't solve is the queue operation. In the cvs head version, thread wake-up is handled by using Ns_CondSignal instead of Ns_CondBroadcast. Signal only wakes up one thread at a time, whereas Broadcast gives every thread a chance to restart execution. In practice what happens is that the thread which called Signal usually gets woken up, so it starts executing again. What can happen is that slow requests get stuck sleeping while one thread grabs quick requests. Essentially you have on thread working the queue. It is then possible for this thread to exit after serving all requests, just as the traffic dies down. You end up with threads waiting to wake up, the driver waiting for new requests and somehow nothing happening. One thing you could do, just for testing if requests are still being processed prior to going into the queue, is to register a prequeue filter. Register it to an exact url which will never be called under normal conditions. When your server gets stuck you can visit the url and see if it logs something (Just use the prequeue filter to issue an ns_log and return filter_ok). But note: never use a prequeue filter in Tcl for anything else. It causes a massive memory leak, probably because a Tcl interp is created and never destroyed. You get about a 1 virtual server sized memory leak per request if your request matches a prequeue filter. If something is logged, it means that the request was processed by the driver thread and probably put into the queue to wait for a free conn thread. If not, it means that something is blocking the driver thread, or maybe something else is going on. tom jackson On Sun, 2009-05-03 at 13:47 -0300, Eduardo Santos wrote: Just another guess: do you have anything between AOLServer and the connections, such as Apache proxy and other things? Most performance problems in AOLServer we've seen are related to the environment, no to the AOLServer itself. 2009/5/1 Dossy Shiobara do...@panoptic.com On 5/1/09 6:17 PM, William Scott Jordan wrote: I think it's worth noting that when AOLserver stops accepting connections, nslog doesn't log any of the failed requests. New connections seem to just be rejected outright. Do you have nscp enabled? Is the control port still responsive to commands? Can you capture the output of join [ns_info threads] \n from the control port once AOLserver stops accepting connections? -- Dossy Shiobara | do...@panoptic.com | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on. (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] nsopenssl client file descriptor issues
Andrew,
Do you have any up-to-date instructions on compiling nsopenssl? For some
reason I'm getting a segfault the instant I try to use the client
ns_httpspost.
I'm think it is related to the linux distribution, but the crash isn't
the random problem you are seeing.
Thanks,
tom jackson
On Thu, 2009-04-30 at 17:59 -0500, Andrew Steets wrote:
Hello,
We recently discovered a problem with the nsopenssl ns_httpsXXX client
commands which was causing SSL close notify alerts (a.k.a. random
binary garbage) to be written to unrelated (non-ssl) file descriptors
in certain cases. While we were trying to come up with a fix, we
stumbled across some other nsopenssl issues.
If you aren't using the nsopenssl *client* functionality this is
probably not interesting. If you aren't interested in hacking the
nsopenssl code then you should realize that this may be a potential
source of frustration. For anyone else, details follow.
All of the ns_https client TCL (https.tcl) commands eventually call
ns_openssl_sockopen to open an SSL connection to a server.
ns_openssl_sockopen, like ns_sockopen, returns two TCL channel ids,
one of which is for reading and the other for writing. The TCL
channels are created in CreateTclChannel() in nsopenssl's tclcmds.c.
The channels are stored in a pair of structs with the following
definition:
typedef struct ChanInfo {
NsOpenSSLConn *sslconn;
SOCKET socket;
Tcl_Channel chan;
void*otherchaninfo;
} ChanInfo;
so the write chaninfo holds a pointer to the read chaninfo and vice
versa. The channels are currently constructed such that the read
channel is associated with the original socket fd created for the ssl
connection, and the write channel is associated with another fd
dup()'ed from the original. They are both associated with the same
NsOpenSSLConn struct, which itself holds the original socket fd as
well.
The channel close function, ChanCloseProc(), has to deal with this two
fd situation, and that is where we run into problems. The close proc
will close the fd associated with whichever channel is being closed,
but will only shutdown the ssl connection when both channels have been
closed.
Here is the slightly edited close chan code:
static int
ChanCloseProc(ClientData arg, Tcl_Interp *interp)
{
ChanInfo *chaninfo = (ChanInfo *) arg;
ChanInfo *otherchaninfo = NULL;
Tcl_UnregisterChannel(interp, chaninfo-chan);
ns_sockclose(chaninfo-socket);
chaninfo-socket = INVALID_SOCKET;
otherchaninfo = (ChanInfo *) chaninfo-otherchaninfo;
if (otherchaninfo-socket == INVALID_SOCKET) {
ns_free(otherchaninfo);
NsOpenSSLConnDestroy(chaninfo-sslconn);
ns_free(chaninfo);
}
return TCL_OK;
}
One problem is that the ns_sockclose() call precedes the
NsOpenSSLConnDestroy() call. NsOpenSSLConnDestroy() calls
SSL_shutdown() on the file descriptor which was previously closed with
ns_sockclose(). SSL_shutdown() tries to write some ssl close notify
messages on the fd. There is no way this can succeed because the fd
was already closed. The error is siliently ignored. Clearly the sock
close needs to come after NsOpenSSLConnDestroy().
But there is more. Now we need to examine two possible cases.
Case 1: The write channel is closed before the read channel. In this
case the dup fd is closed first, and the original FD is closed second.
There is a teensy little race condition here. After the
ns_sockclose() call, the OS may context switch to another thread which
may call open(), dup(), socket() or anything that gets a new FD. It's
also possible that the FD that the OS returns for that call may have
been the one which was previously closed with ns_sockclose(). If we
then switch back to the original thread and call
NsOpenSSLConnDestroy() - SSL_shutdown(), then we will end up writing
and reading on somebody else's file file descriptor! This is
obviously bad, but the chances of this race actually occuring are
probably slim.
Case 2: The read channel is closed before the write channel. This is
the worst. The original fd, the one in the NsOpenSSLConn struct is
closed, but NsOpenSSLConnDestroy is not called because the write
channel is still open and the sslconn * still holds the now invalid
fd. Now we have a much larger window for that FD to be recycled by
the OS and we don't necessarily need an unlikely context switch to be
stung by the race. The following ADP highlights this condition.
%
set fds [ns_openssl_sockopen -nonblock www.att.com 443]
set rfd [lindex $fds 0]
set wfd [lindex $fds 1]
ns_adp_puts rfd: $rfd
ns_adp_puts wfd: $wfd
_ns_https_puts 5 $wfd GET / HTTP/1.0\r
close $rfd
set tmpfd [open /tmp/nsopenssl w]
ns_adp_puts tmpfd: $tmpfd
close $wfd
close $tmpfd
%
If you look at /tmp/nsopenssl after running this ADP, you should see
some random binary garbage
Re: [AOLSERVER] missing access log entries
On Mon, 2009-04-13 at 13:49 -0400, Jim Davidson wrote:
Hi,
A bit old but let me try to be helpful here...
On Apr 3, 2009, at 11:45 PM, Tom Jackson wrote:
Jim,
Thanks for adding some back story. Over the weekend I spent more time
looking into this problem and basically rewrote one portion of ConnRun
and pulled out the auth part of the code into another static function.
As Gustaf noticed, there was a little confusion in the code, I think
mostly related to the use and meaning of the status variable. This
variable has several meanings at different points in ConnRun and it gets
confusing. Plus I wasn't completely sure what was supposed to happen at
each point.
I think I finally figured it out, and it now appears to me that
including the access logging trace where it is makes perfect sense. What
didn't make sense, and was a bug, was the meaning of the value of
status at the point trace filters, server traces and void traces ran.
In effect, status must indicate the success/failure of sending a
response to the client. The current code didn't do that. Here is my
rewrite of the two affected functions, with comments added:
/*
*--
*
* ConnAuthorize --
*
* Try to authorize a connection.
*
* Results:
* NS_OK on successful authorization,
* NS_FILTER_RETURN on authorization failure, or
* NS_ERROR on error.
*
* Side effects:
* Connection request is authorized. On failure an alternative
* response may be sent to the client.
*
*--
*/
static int
ConnAuthorize(Conn *connPtr)
{
Ns_Conn *conn = (Ns_Conn *) connPtr;
NsServer *servPtr = connPtr-servPtr;
intstatus;
status = Ns_AuthorizeRequest(servPtr-server,
connPtr-request-method, connPtr-request-url,
connPtr-authUser, connPtr-authPasswd, connPtr-peer);
switch (status) {
case NS_OK:
break;
case NS_FORBIDDEN:
/*
* NS_OK indicates that a response was sent to the client
* a this point, we must fast-forward to traces,
* so we convert the NS_OK to NS_FILTER_RETURN.
*/
if ((status = Ns_ConnReturnForbidden(conn)) == NS_OK) {
status = NS_FILTER_RETURN;
}
break;
case NS_UNAUTHORIZED:
/*
* NS_OK indicates that a response was sent to the client
* a this point, we must fast-forward to traces,
* so we convert the NS_OK to NS_FILTER_RETURN.
*/
if ((status = Ns_ConnReturnUnauthorized(conn)) == NS_OK) {
status = NS_FILTER_RETURN;
}
break;
case NS_ERROR:
default:
status = NS_ERROR;
break;
}
return status;
}
/*
*--
*
* ConnRun --
*
* Run a valid connection.
*
* Results:
* None.
*
* Side effects:
* Connection request is read and parsed and the cooresponding
* service routine is called.
*
*--
*/
static void
ConnRun(Conn *connPtr)
{
Ns_Conn *conn = (Ns_Conn *) connPtr;
NsServer *servPtr = connPtr-servPtr;
inti, status;
Tcl_Encodingencoding = NULL;
/*
* Initialize the connection encodings and headers.
*/
connPtr-outputheaders = Ns_SetCreate(NULL);
encoding = NsGetInputEncoding(connPtr);
if (encoding == NULL) {
encoding = NsGetOutputEncoding(connPtr);
if (encoding == NULL) {
encoding = connPtr-servPtr-urlEncoding;
}
}
Ns_ConnSetUrlEncoding(conn, encoding);
if (servPtr-opts.hdrcase != Preserve) {
for (i = 0; i Ns_SetSize(connPtr-headers); ++i) {
if (servPtr-opts.hdrcase == ToLower) {
Ns_StrToLower(Ns_SetKey(connPtr-headers, i));
} else {
Ns_StrToUpper(Ns_SetKey(connPtr-headers, i));
}
}
}
/*
* Run the request.
*/
while (1) {
/*
* Proxy requests replace request logic
*/
if (connPtr-request-protocol != NULL
connPtr-request-host != NULL) {
status = NsConnRunProxyRequest((Ns_Conn *) connPtr);
break;
}
/*
* Each stage of request processing can return one of three
* possible results:
* NS_OK means continue to next stage
* NS_FILTER_RETURN means skip to traces
* NS_ERROR means skip to 500 response attempt
*/
status = NsRunFilters(conn, NS_FILTER_PRE_AUTH);
if (status != NS_OK) {
break;
}
status = ConnAuthorize(connPtr);
if (status != NS_OK) {
break;
}
status = NsRunFilters(conn
Re: [AOLSERVER] missing access log entries
You can try the diff link to the parent at the following two links. The second one is older, but each patch is to different files. But please note that my queue.c file (first patch below) is not identical to CVS HEAD, so the patch line numbers will not match up. On Fri, 2009-04-17 at 00:32 +0400, Alexey Pechnikov wrote: http://www.junom.com/gitweb/gitweb.perl?p=aolserver.git;a=commit;h=7ab44 http://www.junom.com/gitweb/gitweb.perl?p=aolserver.git;a=commit;h=20f65 tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] missing access log entries
Gustaf,
Thanks for testing on a production server. I haven't tested this with
the authorization module, but it seems like it should work okay.
The internal server loop bug that I tracked down yesterday led me to
another function which contains code very similar to the new
ConnAuthorize function, so I'm wondering if I should see if I should
maybe export ConnAuthorize to NsConnAuthorize. The overall idea is to
only run Ns_ConnReturnInternalError at the end of the request and not
from inside the authorization or redirection functions (because the
returns Ns_ConnReturnForbidden and Ns_ConnReturnUnauthorized could also
return NS_ERROR.
Any ideas are welcome. (I think just removing Ns_ConnReturnInternalError
from Ns_ConnRedirect will break the loop.)
tom jackson
Here's the two chunks of code:
static int
ConnAuthorize(Conn *connPtr)
{
Ns_Conn *conn = (Ns_Conn *) connPtr;
NsServer *servPtr = connPtr-servPtr;
intstatus;
status = Ns_AuthorizeRequest(servPtr-server,
connPtr-request-method, connPtr-request-url,
connPtr-authUser, connPtr-authPasswd, connPtr-peer);
switch (status) {
case NS_OK:
break;
case NS_FORBIDDEN:
if ((status = Ns_ConnReturnForbidden(conn)) == NS_OK) {
status = NS_FILTER_RETURN;
}
break;
case NS_UNAUTHORIZED:
if ((status = Ns_ConnReturnUnauthorized(conn)) == NS_OK) {
status = NS_FILTER_RETURN;
}
break;
case NS_ERROR:
default:
status = NS_ERROR;
break;
}
return status;
}
and from op.c:
int
Ns_ConnRedirect(Ns_Conn *conn, char *url)
{
Conn *connPtr = (Conn *) conn;
int status;
++connPtr-recursionCount;
/*
* Update the request URL.
*/
Ns_SetRequestUrl(conn-request, url);
/*
* Re-authorize and run the request.
*/
status = Ns_AuthorizeRequest(Ns_ConnServer(conn), conn-request-method,
conn-request-url, conn-authUser,
conn-authPasswd, Ns_ConnPeer(conn));
switch (status) {
case NS_OK:
status = Ns_ConnRunRequest(conn);
break;
case NS_FORBIDDEN:
status = Ns_ConnReturnForbidden(conn);
break;
case NS_UNAUTHORIZED:
status = Ns_ConnReturnUnauthorized(conn);
break;
case NS_ERROR:
default:
status = Ns_ConnReturnInternalError(conn);
break;
}
return status;
}
On Wed, 2009-04-15 at 12:52 +0200, Gustaf Neumann wrote:
your rewrite of the two functions below look fine to me. The structure
is much clearer now, results of the authorization handling are handled
now consistently. I am useing these functions since a few days on one
of my servers and found nothing unusual. Many thanks to Tom!
Andrew, did you test this version as well for your test-cases?
If thise works for you as well (i would assume so), i would think
that version should go into CVS.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
Re: [AOLSERVER] missing access log entries
Jim,
I'm thinking along the same lines. The while {1} thing is similar to a
try block, but it handles the special case where there is really only on
exception, on error or a successful return, fast-forward to
traces...actually check for an error then maybe do traces.
What I'm not sure about is some of the looping which happens with
redirects. Specifically redirecting to non-200 codes. I'm not sure why
there is a need to reauthorize before running client or server error
handlers. In essence there is a chance that the original failure will be
lost in this process and a new failure will replace it.
***Digression***
I seem to remember that you once said it would be nice if there were
more filter points available.
What would be interesting is if you could fast-forward, or rewind to
more points. Right now we have NS_FILTER_BREAK, NS_FILTER_RETURN, NS_OK
and NS_ERROR, which all go to pre-determined points. These really are
not states, just a rough indication of something.
Right now we can rewind to authorization, but then we skip post-auth
filters and run the new request.
Anyway, it seems like the while loop could change to some kind of
generic state machine.
Instead of a fixed set of steps, each step could potentially end with a
restart with a new set of steps, or just a restart of the same set. You
could use this to avoid running through the list of filters for urls
which will never match the current url/method.
Maybe I can do a mock-up in Tcl.
***End Digression***
Anyway, here is where I'm at with the error loop:
After a few additional changes I have got one thing working: all errors
are handled after the while {1} loop. I have one remaining problem with
Ns_ConnReturnInternalError. It looks like this:
Ns_ConnReturnInternalError(Ns_Conn *conn)
{
int result;
Ns_SetTrunc(conn-outputheaders, 0);
if (ReturnRedirect(conn, 500, result)) {
return result;
}
return Ns_ConnReturnNotice(conn, 500, Server Error,
The requested URL cannot be accessed
due to a system error on this server.);
}
The problem is that the result is returned base upon the existence of
a redirect mapping. This works for everything except errors which take
place when trying to run the configured error handler.
This case highlights exactly what you are talking about. The result is
an error, but ReturnRedirect found a handler. Why did the handler fail?
In the bug I'm working on, the request line is PROPFIND /abc HTTP/1.1
It seems like the correct error is a 501 Not Implemented, but that
error should have shown up before the 404 redirect.
I'm thinking the problem is in ReturnRedirect and how redirect are
mapped (they leave out the method). Adding a method in the redirect
mapping would be interesting, allowing different handlers for different
methods.
The other thing I'm looking at is trying to short circuit the redirect
code so that it doesn't redirect to the same url.
tom jackson
On Wed, 2009-04-15 at 13:55 -0400, Jim Davidson wrote:
I'm wondering if there needs to be some more specific exception flags
stored in the connection structure to handle all these cases and some
C and Tcl api's to access/modify the same. It seems you're doing the
good work of rationalizing all the error conditions that had been
pretty confused together and shuffling / updating the code to handle
the various cases but more explicit tracking of the error conditions
(execution, pre/post filter, connection, auth, etc.) could make things
easier.This may lead the code to look like more try/catch type
exception handling instead of a non-obvious code path understandable
only with the surrounding comments.
-Jim
On Apr 15, 2009, at 11:12 AM, Tom Jackson wrote:
Gustaf,
Thanks for testing on a production server. I haven't tested this with
the authorization module, but it seems like it should work okay.
The internal server loop bug that I tracked down yesterday led me to
another function which contains code very similar to the new
ConnAuthorize function, so I'm wondering if I should see if I should
maybe export ConnAuthorize to NsConnAuthorize. The overall idea is to
only run Ns_ConnReturnInternalError at the end of the request and not
from inside the authorization or redirection functions (because the
returns Ns_ConnReturnForbidden and Ns_ConnReturnUnauthorized could
also
return NS_ERROR.
Any ideas are welcome. (I think just removing
Ns_ConnReturnInternalError
from Ns_ConnRedirect will break the loop.)
tom jackson
Here's the two chunks of code:
static int
ConnAuthorize(Conn *connPtr)
{
Ns_Conn*conn = (Ns_Conn *) connPtr;
NsServer *servPtr = connPtr-servPtr;
intstatus;
status = Ns_AuthorizeRequest(servPtr-server,
connPtr-request-method, connPtr-request-url,
connPtr-authUser, connPtr-authPasswd, connPtr-peer);
switch (status) {
case
[AOLSERVER] Tracked down bug with PROPFIND / OPTIONS methods
Over the last few years some users have noticed that their servers suddenly stop responding, and the error log has entries similar to this: [-conn:965-] Error: return: failed to redirect 'PROPFIND /global/file-not-found.tcl': exceeded recursion limit of 3 [-conn:965-] Error: return: failed to redirect 'PROPFIND /global/server-error.tcl': exceeded recursion limit of 3 The second error.log line then repeats hundreds or thousands of times until the server stops responding. The question is why this happens, and what do these log entries tell us. I think I have found out the answer, but the fix isn't apparent. The first error message indicates the recursion limit code is working correctly, after three tries, the HTTP status code goes from 404 to 500. The second error message indicates a similar recursion limit is reached looking for a 500 handler. Unfortunately there is a loop here: Ns_ConnReturnInternalError executes ReturnRedirect which executes Ns_ConnRedirect which executes Ns_ConnReturnInternalError This loop accounts for the remaining error log entries. One problem is that ReturnRedirect uses the redirects configured like this: ns_section ns/server/farid/redirects ns_param 404 /fnf-tmpl.tcl ns_param 403 global/forbidden.html ns_param 500 global/server-error.tcl What is missing, it seems to me is the method of the request. The method is used in Ns_AuthorizeRequest and Ns_ConnRunRequest. The quick fix is to not configure a 500 redirect. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] missing access log entries
On Mon, 2009-04-13 at 13:49 -0400, Jim Davidson wrote: On Apr 3, 2009, at 11:45 PM, Tom Jackson wrote: * Filters seem uniquely superior when the url pattern determines which code should run. Unfortunately filters are many times registered to / *, the global pattern. This is not a filter in the typical sense. The only benefit is the ability to stage code to run in a particular order at a particular stage. The question is if there should be a separate stage for global filters, or a separate mechanism to register global code which runs at certain point, maybe conditioned on previous events. This is a like a generalized ns_atclose. I think a good idea gets implemented, bad ideas linger to avoid breaking things so that's why both exist. BTW: I remembered why the newer connection cleanup stuff was added -- it was designed to interact with the connection local storage API which can be accessed in the driver thread, ostensibly to fetch and store network-related assets in an efficient manner. The model is similar to Pthreads local storage and thus needed non-conditional cleanup callbacks. I should point out a mistake I made in the above analysis. The access logging filter is a global filter. It actually works exactly like I was proposing. Until I actually looked at the nslog module, I assumed it was registered as a filter. It isn't. It is a server trace A server trace is identical to a cleanup trace, it just runs at a different point, and is based upon successfully sending a response to the client. This success is indicated/protected by status = NS_OK||NS_FILTER_RETURN. I experimented with using a cleanup trace in nslog.c, and it worked exactly the same. This required the following adjustment: Ns_RegisterServerTrace(server, LogTrace, logPtr); changed to: Ns_RegisterConnCleanup(server, LogTrace, logPtr); An important distinction between this API and the filter API is that an interp is not required. You can run generic C code. There is also no url pattern matching and all traces/cleanups run. * The access logging module is ancient, simple and in desperate need of professional help. Agreed. And, google analytics and chartbeat.com are pretty cool :) Anyone have ideas on using these? * The Internal redirecting functionality is poorly documented and understood. These are configured as proxies or redirects. Yes -- it was always a great disappointment for me :( Yeah, it seems like you could use it for lots of things, like a generic FSM engine. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] More on caching content...
Go to: http://www.junom.com/gitweb/gitweb.perl?p=aolserver.git It has a grep function built into every page, works great for exactly this case...looking for all instances of a text string. But it also presents a hyperlink result page. For old versions of the code, it seems like the pickaxe feature is helpful. Note: use the top level directory for searches and make sure you are useing grep or pickaxe if you get no results. I doubt there is anything which specifically sets cache-control in aolserver. tom jackson On Mon, 2009-04-13 at 14:47 -0500, David Walker wrote: To aid your troubleshooting, I would recommend a grep or other full-text search of the code running on AOLServer to determine if there are any circumstances where it would generate a cache-control header. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] missing access log entries
Gustaf,
Yep, you are right. Looking at this again, I noticed a few additional
things which might help get this working as expected.
For one, the log trace is registered with Ns_RegisterServerTrace. The
documentation with this function says:
* Register a connection trace procedure. Traces registered
* with this procedure are only called in FIFO order if the
* connection request procedure successfully responds to the
* clients request.
I read this to include successfully sending a forbidden, unauthorized or
internal server error response (Otherwise the access log would contain
only 200 response codes).
So I have rearranged the code so that the status code reflects the
actual status of sending a response to the client.
Attached is a patch based upon the original code, not my last attempt,
and more info at:
http://www.junom.com/gitweb/gitweb.perl?p=aolserver.git;a=commit;h=7ab44
tom jackson
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
diff --git a/aolserver/nsd/queue.c b/aolserver/nsd/queue.c
index c620393..051abdc 100644
--- a/aolserver/nsd/queue.c
+++ b/aolserver/nsd/queue.c
@@ -56,6 +56,7 @@ typedef struct ConnData {
*/
static void ConnRun(Conn *connPtr); /* Connection run routine. */
+static int ConnAuthorize(Conn *connPtr); /* Connection Authorize routine. */
static void AppendConnList(Tcl_DString *dsPtr, Conn *firstPtr, char *state);
/*
@@ -533,6 +534,59 @@ NsConnThread(void *arg)
/*
*--
*
+ * ConnAuthorize --
+ *
+ * Try to authorize a connection.
+ *
+ * Results:
+ * NS_OK on successful authorization,
+ * NS_FILTER_RETURN on failure, or
+ * NS_ERROR on error.
+ *
+ * Side effects:
+ * Connection request is authorized. On failure an alternative
+ * response may be sent to the client.
+ *
+ *--
+ */
+
+static int
+ConnAuthorize(Conn *connPtr)
+{
+Ns_Conn *conn = (Ns_Conn *) connPtr;
+NsServer *servPtr = connPtr-servPtr;
+intstatus;
+
+status = Ns_AuthorizeRequest(servPtr-server,
+ connPtr-request-method, connPtr-request-url,
+ connPtr-authUser, connPtr-authPasswd, connPtr-peer);
+
+switch (status) {
+case NS_OK:
+break;
+case NS_FORBIDDEN:
+if ((status = Ns_ConnReturnForbidden(conn)) == NS_OK) {
+ status = NS_FILTER_RETURN;
+}
+break;
+case NS_UNAUTHORIZED:
+if ((status = Ns_ConnReturnUnauthorized(conn)) == NS_OK) {
+status = NS_FILTER_RETURN;
+}
+break;
+case NS_ERROR:
+default:
+status = NS_ERROR;
+break;
+}
+
+return status;
+}
+
+
+/*
+ *--
+ *
* ConnRun --
*
* Run a valid connection.
@@ -582,52 +636,49 @@ ConnRun(Conn *connPtr)
* Run the request.
*/
-if (connPtr-request-protocol != NULL connPtr-request-host != NULL) {
- status = NsConnRunProxyRequest((Ns_Conn *) connPtr);
-} else {
- status = NsRunFilters(conn, NS_FILTER_PRE_AUTH);
- if (status == NS_OK) {
- status = Ns_AuthorizeRequest(servPtr-server,
- connPtr-request-method, connPtr-request-url,
- connPtr-authUser, connPtr-authPasswd, connPtr-peer);
- switch (status) {
- case NS_OK:
- status = NsRunFilters(conn, NS_FILTER_POST_AUTH);
- if (status == NS_OK) {
- status = Ns_ConnRunRequest(conn);
- }
- break;
-
- case NS_FORBIDDEN:
- Ns_ConnReturnForbidden(conn);
- break;
-
- case NS_UNAUTHORIZED:
- Ns_ConnReturnUnauthorized(conn);
- break;
-
- case NS_ERROR:
- default:
- Ns_ConnReturnInternalError(conn);
- break;
- }
-} else if (status != NS_FILTER_RETURN) {
-/* if not ok or filter_return, then the pre-auth filter coughed
- * an error. We are not going to proceed, but also we
- * can't count on the filter to have sent a response
- * back to the client. So, send an error response.
- */
-Ns_ConnReturnInternalError(conn);
-status = NS_FILTER_RETURN; /* to allow tracing to happen */
+while (1) {
+
+if (connPtr-request-protocol != NULL
+ connPtr-request-host != NULL) {
+
+status = NsConnRunProxyRequest((Ns_Conn *) connPtr);
+break;
+}
+
+status = NsRunFilters(conn, NS_FILTER_PRE_AUTH);
+if (status != NS_OK) {
+break;
+}
+
+status = ConnAuthorize(connPtr);
+if (status != NS_OK) {
+break;
}
+
+status = NsRunFilters(conn, NS_FILTER_POST_AUTH);
+if (status != NS_OK) {
+break
Re: [AOLSERVER] missing access log entries
that the conn is closed and just log an error. I have not checked if the access.log entry would change to a 500 response in this case. That would be somewhat misleading, but it might be helpful in finding errors related to specific page requests. Finish blowhard answer Sorry for the long winded response, I'll put short answers below: On Sat, 2009-04-11 at 03:53 +0200, Gustaf Neumann wrote: Tom Jackson schrieb: Okay, great! The changes should probably be divided into two patches: good idea. i was not able to follow the full discussion and most likely one needs more thoughts and largers test cases to consider. Do i understand correctly: - when either the POST_AUTH filter or the request returns != NS_OK the server returns a 500 error - if there was an earlier ns_return XXX, it is ignored. ConnRun doesn't know anything about earlier run code, just the result code was not NS_OK or NS_FILTER_RETURN. NsRunFilters retuns either NS_OK or NS_FILTER_BREAK, NS_FILTER_TRACE or NS_FILTER_RETURN. Don't we have to deal with the other non-ok cases? No, filters can only return NS_OK, NS_FILTER_BREAK or NS_FILTER_RETURN. More filters and the traces are run, when the status is NS_OK or NS_FILTER_RETURN. If i look at the code, i wonder, what happens with filters and traces for status codes like NS_FORBIDDEN, NS_UNAUTHORIZED, or NS_SHUTDOWN? An error occurs, filters must return NS_OK, NS_FILTER_BREAK or NS_FILTER_RETURN. Have you checked what happens, when a Ns_ConnRunRequest ended in Tcl with a return, break or continue? [return] is okay, [break] and [continue] will be errors if code reaches the end of a proc or the Tcl top level outside of a loop (usually this happens pretty quick and is a Tcl use bug). Just notice here that [return] has a return code of TCL_OK. Also notice that Ns_ConnRunRequest can only return NS_OK or NS_ERROR, no other options are needed here. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] missing access log entries
Andrew, Okay, great! The changes should probably be divided into two patches: queue.c is a behavior change and doesn't address any bugs, just returns a 500 response if possible, and allows trace filters to run. My guess is that this patch is backwards compatible with all code because someone would have noticed by now that trace filters don't always run (assuming their code relied on them running). If they noticed that trace filters were not running, then they would have someone adapted their code to the known behavior, so their code should still work. In addition, it is now dangerous to rely on code to initialize in a pre/postauth filter and then run in a trace filter because the request url may have changed, which would alter what trace filters run, unless they are registered to the global pattern /*. In that case, you can now use ns_ictl to run traces with/without a conn (you shouldn't need a conn in a trace except for logging about the conn). ns_atclose might work in some cases, it might still have info about the conn. The other patch is to adprequest.c and adpeval.c, which is a pure bug-fix. It only restores the expected behavior of ns_adp_return, ns_adp_break and ns_adp_abort, as well as returning the previously documented AOLserver request return codes NS_OK or NS_ERROR. Unless there are additional questions about the bugfix, I'll checkout the HEAD version and commit a patch. The patch to queue.c needs at least a little community feedback, and should also include a changelog and maybe some documentation changes. tom jackson On Thu, 2009-04-09 at 20:10 -0500, Andrew Steets wrote: Hi Tom, sorry to go dark for so long. It was operator error. I was in a hurry and I don't think I restarted the server after I installed the patched version of the server. I checked again just now and everything works as expected. -Andrew On Mon, Apr 6, 2009 at 4:14 PM, Tom Jackson t...@rmadilo.com wrote: Andrew, Hmmm, well without knowing how you tested this, I can't help much. I created a few test adp pages. I tested them before my changes to identify the problems. Here is an example set of pages: include.adp: % ns_adp_puts before include ns_adp_include test-ns-return.adp ns_adp_puts after include ns_log Notice finished include.adp % test-ns-return.adp: % ns_return 200 text/plain hi ns_adp_abort ns_log Notice test-ns-return.adp after ns_adp_abort % The error.log should contain neither of the Notice logs. The access.log should have a 200 response of content length 2. Even this produces an access.log entry: % ns_adp_puts hi ns_adp_abort % A zero length 200 response: 127.0.0.1 - - [06/Apr/2009:13:54:52 -0700] GET /just-abort.adp HTTP/1.1 200 0 Did you patch the other two files? (Note that my queue.c file is not identical to yours, so the patch needs to be applied by hand I think.) queue.c handles changes to allow logging during error conditions adprequest.c changes allows distinguishing between actual errors and adp signaling and translates Tcl return codes into AOLserver request return codes. Changes also ensure that the adp buffer is cleaned up in all cases. adpeval.c changes just ensure that the tcl error code is set to correspond to the ADP exception code. The code probably needs a comment because actual errors in ADP processing is signaled when the Tcl return code is TCL_OK and the adp exception code is ADP_OK. Why? Because on a tcl error, the ADP code doesn't get to change adp.exception to something else. The bugs in the current code were due to the awkward but necessary maintenance of these two return codes. The ADP code has gone through a lot of significant changes, so it is easy to see how these details didn't make it through correctly. But the simplicity of fixing them indicates that the code is in pretty good shape. Anyway, that is the bugs. The code in queue.c is not bug related, but allows the client to receive a 500 response on error and to allow logging during error conditions. Here is a link to my patch to my code: http://www.junom.com/gitweb/gitweb.perl?p=aolserver.git;a=commit;h=ca26f1a tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] Caching questions
On Tue, 2009-04-07 at 09:48 -0700, Jade Rubick wrote: Why not use this algorithm: 1) check if you have local cached copy, and if the date on it is within xx hours 2) if available, return it. 3) if not, retrieve remote copy. The first critical question is if you can use caching. You can only use it when it is okay to serve up old data, or to serve the same data to multiple clients. The next question, assuming stale, or generic data is okay, is how often should the cache refresh from the master source. But as Jade points out, the decision must be made at the level of the cache, not by comparing sourrce to cache. There is another alternative. You can create an complete cache, a complete hard copy. Then you can push updates to the cache. In this case, the cache is always served and the update responsibility is offloaded to the master. That could be difficult if you can't predict all the changes which will result from a particular update. But you actually have the same issue either way, which it is usually difficult to cache dynamic applications. Bottom line is that you delegate to the cache an acceptable staleness/general applicability. But you can't expect a cache to catch dynamic updates. tom jackson -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] missing access log entries
The attached patch fixes ns_adp_break, it differs from the previous
patch by one line in adpeval.c
tom jackson
On Sat, 2009-04-04 at 16:25 -0500, Andrew Steets wrote:
Hi Tom,
Attachments seem to work ok on this list.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
t...@boron:~/cvs/aolserver/aolserver/nsd$ diff -u adpeval.c /web/tnt/aolserver/aolserver/nsd/adpeval.c
--- adpeval.c 2008-06-10 17:23:02.0 -0700
+++ /web/tnt/aolserver/aolserver/nsd/adpeval.c 2009-04-05 10:43:01.0 -0700
@@ -1017,7 +1017,12 @@
break;
case ADP_RETURN:
itPtr-adp.exception = ADP_OK;
- /* FALLTHROUGH */
+ result = TCL_OK;
+ break;
+case ADP_ABORT:
+case ADP_BREAK:
+result = TCL_ERROR;
+break;
default:
result = TCL_OK;
break;
t...@boron:~/cvs/aolserver/aolserver/nsd$ diff -u adprequest.c /web/tnt/aolserver/aolserver/nsd/adprequest.c
--- adprequest.c 2008-06-10 17:23:02.0 -0700
+++ /web/tnt/aolserver/aolserver/nsd/adprequest.c 2009-04-04 09:33:00.0 -0700
@@ -154,9 +154,15 @@
result = NsAdpInclude(itPtr, 2, objv, start, ttlPtr);
Tcl_DecrRefCount(objv[0]);
Tcl_DecrRefCount(objv[1]);
-if (NsAdpFlush(itPtr, 0) != TCL_OK || result != TCL_OK) {
- return NS_ERROR;
+if ((result != TCL_OK) (itPtr-adp.exception == ADP_OK)) {
+ itPtr-adp.exception = ADP_ABORT;
+ NsAdpFlush(itPtr, 0);
+ return NS_ERROR;
}
+if (NsAdpFlush(itPtr, 0) != TCL_OK) {
+ return NS_ERROR;
+}
+
return NS_OK;
}
@@ -216,6 +222,7 @@
Tcl_ResetResult(interp);
if (itPtr-adp.exception == ADP_ABORT) {
Tcl_SetResult(interp, adp flush disabled: adp aborted, TCL_STATIC);
+ result = TCL_OK;
} else if (len == 0 stream) {
result = TCL_OK;
} else {
t...@boron:~/cvs/aolserver/aolserver/nsd$ diff -u queue.c /web/tnt/aolserver/aolserver/nsd/queue.c
--- queue.c 2009-04-04 08:43:56.0 -0700
+++ /web/tnt/aolserver/aolserver/nsd/queue.c 2009-04-03 16:17:46.0 -0700
@@ -595,6 +595,13 @@
status = NsRunFilters(conn, NS_FILTER_POST_AUTH);
if (status == NS_OK) {
status = Ns_ConnRunRequest(conn);
+ if (status != NS_OK) {
+ Ns_ConnReturnInternalError(conn);
+ status = NS_OK; /* to allow tracing to happen */
+ }
+ } else {
+ Ns_ConnReturnInternalError(conn);
+ status = NS_FILTER_RETURN; /* to allow tracing to happen */
}
break;
Re: [AOLSERVER] missing access log entries
Here is a test patch for the ns_adp_abort issue.
The patch enables sending an error message in the case of an actual
error during adp processing, or after a postauth filter (preauth errors
already allow this behavior).
Also, logging is enabled in all cases. If an error occurs, a 500
response is sent and this is what is logged.
I haven't tested this with ns_adp_break. But it works with ns_adp_return
and ns_adp_abort as well as error handling in and adp.
tom jackson
Not sure if I can attach the patch, but here goes:
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to
lists...@listserv.aol.com with the
body of SIGNOFF AOLSERVER in the email message. You can leave the Subject:
field of your email blank.
t...@boron:~/cvs/aolserver/aolserver/nsd$ diff -u adpeval.c /web/tnt/aolserver/aolserver/nsd/adpeval.c
--- adpeval.c 2008-06-10 17:23:02.0 -0700
+++ /web/tnt/aolserver/aolserver/nsd/adpeval.c 2009-04-04 09:16:35.0 -0700
@@ -1017,7 +1017,11 @@
break;
case ADP_RETURN:
itPtr-adp.exception = ADP_OK;
- /* FALLTHROUGH */
+ result = TCL_OK;
+ break;
+case ADP_ABORT:
+result = TCL_ERROR;
+break;
default:
result = TCL_OK;
break;
t...@boron:~/cvs/aolserver/aolserver/nsd$ diff -u adprequest.c /web/tnt/aolserver/aolserver/nsd/adprequest.c
--- adprequest.c 2008-06-10 17:23:02.0 -0700
+++ /web/tnt/aolserver/aolserver/nsd/adprequest.c 2009-04-04 09:33:00.0 -0700
@@ -154,9 +154,15 @@
result = NsAdpInclude(itPtr, 2, objv, start, ttlPtr);
Tcl_DecrRefCount(objv[0]);
Tcl_DecrRefCount(objv[1]);
-if (NsAdpFlush(itPtr, 0) != TCL_OK || result != TCL_OK) {
- return NS_ERROR;
+if ((result != TCL_OK) (itPtr-adp.exception == ADP_OK)) {
+ itPtr-adp.exception = ADP_ABORT;
+ NsAdpFlush(itPtr, 0);
+ return NS_ERROR;
}
+if (NsAdpFlush(itPtr, 0) != TCL_OK) {
+ return NS_ERROR;
+}
+
return NS_OK;
}
@@ -216,6 +222,7 @@
Tcl_ResetResult(interp);
if (itPtr-adp.exception == ADP_ABORT) {
Tcl_SetResult(interp, adp flush disabled: adp aborted, TCL_STATIC);
+ result = TCL_OK;
} else if (len == 0 stream) {
result = TCL_OK;
} else {
t...@boron:~/cvs/aolserver/aolserver/nsd$ diff -u queue.c /web/tnt/aolserver/aolserver/nsd/queue.c
--- queue.c 2009-04-04 08:43:56.0 -0700
+++ /web/tnt/aolserver/aolserver/nsd/queue.c 2009-04-03 16:17:46.0 -0700
@@ -595,6 +595,13 @@
status = NsRunFilters(conn, NS_FILTER_POST_AUTH);
if (status == NS_OK) {
status = Ns_ConnRunRequest(conn);
+ if (status != NS_OK) {
+ Ns_ConnReturnInternalError(conn);
+ status = NS_OK; /* to allow tracing to happen */
+ }
+ } else {
+ Ns_ConnReturnInternalError(conn);
+ status = NS_FILTER_RETURN; /* to allow tracing to happen */
}
break;
Re: [AOLSERVER] missing access log entries
Andrew, If the connection has been closed, nothing will be sent. But anyone who uses streaming output will have to live with the possibility that an error will occur mid-response. I don't recommend using streaming output for many reasons, but the most obvious is the one you mention: you (the generic developer) made a mistake in assuming that the response would complete without error. HTTP supports chunked output to handle situations where an entire response could never fit into memory. Anyway, there are many useless rules in the HTTP standards, but the basic signaling of what happened and maybe why is not in this category. Beyond returning information to the client, it is also useful to log the correct response code. My second to last post included a long list of observations about AOLserver code. One of these observations is that the AOLserver API includes a number of API which seem easy to use but actually require a lot of care in application. The entire ADP API falls into this category and also extends to ADP configuration options. What that means is that even with bug free API code, application developers must do lots of experimentation to verify how stuff actually works. The patch is not an attempt to change the basic algorithm of request processing. There were a few bugs which allowed confusion of actual errors and adp signaling. And there was the original behavior of skipping trace filters, including logging, when an error occurred, and this behavior has changed over time. My patch fixes a few bugs and applies the same error handling to all errors, regardless of where they occur. One other note: the ns_adp_ctl seems to have some bugs. Errors are only logged once (after calling ns_adp_ctl stricterror 1), and then the error logging flag seems to remain in place forever. It seems that this flag should be reset for each request, otherwise errors are not logged. tom jackson On Sat, 2009-04-04 at 16:25 -0500, Andrew Steets wrote: Hi Tom, Attachments seem to work ok on this list. I don't think we can return 500 internal server error after Ns_ConnRunRequest has been invoked as it may have already sent an http response code via streaming output or ns_returnxxx. -Andrew On Sat, Apr 4, 2009 at 12:00 PM, Tom Jackson t...@rmadilo.com wrote: Here is a test patch for the ns_adp_abort issue. The patch enables sending an error message in the case of an actual error during adp processing, or after a postauth filter (preauth errors already allow this behavior). Also, logging is enabled in all cases. If an error occurs, a 500 response is sent and this is what is logged. I haven't tested this with ns_adp_break. But it works with ns_adp_return and ns_adp_abort as well as error handling in and adp. tom jackson Not sure if I can attach the patch, but here goes: -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to lists...@listserv.aol.com with the body of SIGNOFF AOLSERVER in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] missing access log entries
Andrew, I wasted a little more time looking at the actual code. My impression is that everything is working as expected. If there is an error in a postauth filter or in adp processing (registered proc), trace filters are skipped. Until about 4.5, errors during preauth also skipped trace filters. Not sure why this change was made. The only think that matters is what happens in Ns_AdpRequest. If there were no errors, the request will be logged. In order to get ns_adp_abort to work correctly, the tcl result must be set to TCL_ERROR until code returns to Ns_AdpRequest. This is why an additional structure is maintained for the adp exception, which is independent of the tcl exception. In this case, adp.exception indicates what actually happened during adp processing. So things seem to be working as intended, and they have been working the same way for a long time. It might be possible that you are misusing ns_adp_abort, or something else is messing up. Could you provide a simple test case, probably a few nested adp includes, which repeats the issue? Without a test case of what you think should work differently, it is hard to give any more advice. In general, when an error occurs during a request, the response is by definition an error response, so the original request might get transformed into an internal redirect to your error handling page. An error in this page, or a missing error page could cause further problems. Bottom line: no reason to believe that this is a bug. tom jackson p.s. this case seems to validate my belief that the hardest bug to find and fix is one that doesn't actually exist. On Thu, 2009-04-02 at 18:03 -0500, Andrew Steets wrote: What was the original purpose of trace filters? At the C API level there is a distinction between between a trace filter and a cleanup callback, but it doesn't look like you can register a cleanup proc from TCL. Maybe this was mistakenly omitted? The cleanup procs run unconditionally. It seems like that is the most appropriate place to handle cleanup of resources. Alternatively we could change the trace filters to run regardless of the Ns_ConnRunRequest() return status, but then that would make them basically the same as the cleanups. I looked a little deeper into the source. The confusion seems to arise in NsAdpFlush() which is run at the end of all ADP processing. The code there is smart enough to recognize when an abort exception has been signalled; it sets the TCL result to adp flush disabled: adp aborted, but it still returns TCL_ERROR. That is essentially where the TCL exception gets turned into a full blown connection processing error. We could change NsAdpFlush() to return success when it recognizes the abort exception, or just not run NsAdpFlush() for abort exceptions. There would still be cases where trace filters would not run though. For instance if you called ns_returnxxx without calling ns_adp_abort. I'm not sure if that is a bad thing. It would be nice to hear from anyone who knows about the original motivation for the trace and cleanup filters. -Andrew On Thu, Apr 2, 2009 at 3:53 PM, Tom Jackson t...@rmadilo.com wrote: Gustaf, You may be using traces but not realize it, it sounds like ns_adp_abort isn't don't what was originally intended. I wouldn't worry about an runtime error caused during running traces, it would be an error to even use ns_adp_abort in a trace filter because the connection is already finished. This is analogous to calling [break] outside of a loop. It seems important to consider ns_adp_abort, ns_adp_return and ns_adp_break as a unit. They add necessary loop type controls so that developers can create deeply nested code and still get out of it without the need to use [catch]. But, like a lot of AOLserver specific procedures, there is no hand-holding in their use. They can be misued. In this particular case, it looks like somewhere along the way, ns_adp_abort was modified to not work as expected. The desired effect is exactly what you would get by returning filter_return from a preauth or postauth filter. This effect is to skip to trace filters, not past them. Skipping trace filters even on an aborted connection would be a disaster for any application which relies on cleanup of resources. tom jackson On Thu, 2009-04-02 at 11:12 +0200, Gustaf Neumann wrote: Andrew Steets schrieb: The patch I sent earlier seems to fulfill these needs, but I am worried about corner cases where LogTrace (from the nslog module) could blow up. Nothing about the state of the Conn
Re: [AOLSERVER] missing access log entries
Hey,
Hopefully this is my last post on this subject, I think I actually found
the bug.
The bug is in NsAdpFlush from nsd/adprequest.c:
214- */
215-
216-Tcl_ResetResult(interp);
217:if (itPtr-adp.exception == ADP_ABORT) {
218-Tcl_SetResult(interp, adp flush disabled: adp aborted,
TCL_STATIC);
219-result = TCL_OK;
220-} else if (len == 0 stream) {
The bug was a missing line setting result to TCL_OK. (line 219).
Also, ns_adp_return cannot be used after and ns_returnxxx command as adp
processing continues after calling it.
Here are two test files:
test-adp-abort.adp:
%
ns_return 200 text/plain hi
ns_adp_abort
%
test-adp-return.adp:
%
ns_adp_puts hi
ns_adp_return
%
Both of these result in an access.log entry.
Before the change, ns_adp_abort would lead to an error message:
adp flush failed: connection closed
abort exception raised
while processing connection #2:
GET /test.adp HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: ...
Accept:
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cache-Control: max-age=0
This error message is valid if ns_adp_return is used after and
ns_returnxxx.
tom jackson
On Fri, 2009-04-03 at 08:33 -0700, Tom Jackson wrote:
Andrew,
I wasted a little more time looking at the actual code. My impression is
that everything is working as expected. If there is an error in a
postauth filter or in adp processing (registered proc), trace filters
are skipped. Until about 4.5, errors during preauth also skipped trace
filters. Not sure why this change was made.
The only think that matters is what happens in Ns_AdpRequest. If there
were no errors, the request will be logged. In order to get ns_adp_abort
to work correctly, the tcl result must be set to TCL_ERROR until code
returns to Ns_AdpRequest. This is why an additional structure is
maintained for the adp exception, which is independent of the tcl
exception. In this case, adp.exception indicates what actually happened
during adp processing.
So things seem to be working as intended, and they have been working the
same way for a long time. It might be possible that you are misusing
ns_adp_abort, or something else is messing up.
Could you provide a simple test case, probably a few nested adp
includes, which repeats the issue? Without a test case of what you think
should work differently, it is hard to give any more advice.
In general, when an error occurs during a request, the response is by
definition an error response, so the original request might get
transformed into an internal redirect to your error handling page. An
error in this page, or a missing error page could cause further
problems.
Bottom line: no reason to believe that this is a bug.
tom jackson
p.s. this case seems to validate my belief that the hardest bug to find
and fix is one that doesn't actually exist.
On Thu, 2009-04-02 at 18:03 -0500, Andrew Steets wrote:
What was the original purpose of trace filters? At the C API level
there is a distinction between between a trace filter and a cleanup
callback, but it doesn't look like you can register a cleanup proc
from TCL. Maybe this was mistakenly omitted?
The cleanup procs run unconditionally. It seems like that is the most
appropriate place to handle cleanup of resources. Alternatively we
could change the trace filters to run regardless of the
Ns_ConnRunRequest() return status, but then that would make them
basically the same as the cleanups.
I looked a little deeper into the source. The confusion seems to
arise in NsAdpFlush() which is run at the end of all ADP processing.
The code there is smart enough to recognize when an abort exception
has been signalled; it sets the TCL result to adp flush disabled: adp
aborted, but it still returns TCL_ERROR. That is essentially where
the TCL exception gets turned into a full blown connection processing
error. We could change NsAdpFlush() to return success when it
recognizes the abort exception, or just not run NsAdpFlush() for abort
exceptions.
There would still be cases where trace filters would not run though.
For instance if you called ns_returnxxx without calling ns_adp_abort.
I'm not sure if that is a bad thing.
It would be nice to hear from anyone who knows about the original
motivation for the trace and cleanup filters.
-Andrew
On Thu, Apr 2, 2009 at 3:53 PM, Tom Jackson t...@rmadilo.com wrote:
Gustaf,
You may be using traces but not realize it, it sounds like
ns_adp_abort isn't don't what was originally intended.
I wouldn't worry about an runtime error caused during running
traces, it
would be an error to even use ns_adp_abort in a trace
