Re: Apache2 + Mod Perl 2.0 + Template Toolkit
hi :) On Tue, Sep 21, 2010 at 5:24 AM, Chris Ray wrote: > Hello, I'm currently attempting to update an Apache Cluster from Apache 1.3 > to Apache2. The website served by this cluster uses Template Toolkit > extensively. It is my understanding that the original Apache::Template > module doesn't work with Mod Perl 2.0 and that you had started porting a > version of this plug-in that does? > > > > Did you finish this? if so where can I get hold of it? you should definitely ask the mod_perl mailing list about this - I haven't worked on that in quite a long time. all that said, what I did do is here http://www.modperlcookbook.org/~geoff/modules/experimental/Apache-Template-2.00_01.tar.gz though I have absolutely no idea whether it works for the current version of TT or not. --Geoff
Re: Strange process behavior
Are you getting "variable will not stay shared" messages in your error_log? If you are, you might be using a lexical variable outside of the function it was defined in. When this happens, you'll get a closure, which is a useful tool sometimes, but for us regular people it means that old data will unexpectedly stay in the variables from one request to the other. The fix is to make sure that every variable (that is not meant to be global) is declared as "my" inside any sub where it is used. http://modperlbook.org/html/22-2-1-Value-of-x-will-not-stay-shared-at-line-5.html -Tuomo On Mon, Sep 20, 2010 at 11:52:50AM -0400, Sharon Pattison wrote: > Hi, > > > We are experiencing a strange problem on our web servers for our site, and > are not sure if the problem is mod_perl related, but thought we would turn > here for help in case someone else has experienced this issue. > Unfortunately, we are not sure exactly when the problem started, so we can’t > point to any particular change that might have prompted it. > > > > We are using Apache 1.3.42 with mod_perl 1.31 and Perl 5.8.3 on Solaris 9. > What is happening is that a particular Apache child process starts having an > issue where the wrong content is being returned for a request. The content > returned is always from the correct script, but it has the wrong data. For > example, if we request a product detail page for a particular product, we > might get a different product, or we might get a message that the product > does not exist, when it does. > > > > After adding a lot of logging to our logs, we discovered that the Apache > request object will have the correct information, but the CGI object never > seems to get the global values reset, so is always has the query_string of > the request where things started to go wrong. The process will continue to > get and respond to requests, but the CGI object’s query_string never > changes, so the user either gets no information or the wrong information. > > > > For example, if we had a request for mysite.com/product?sku=12345, and the > process that served this request starts to have the problem, then we see > that for every request that process gets, logging $q->query_string gives us > ‘sku=12345’. It seems like the initialize_globals() CGI method is never > being executed in the process cleanup. > > > > There doesn’t seem to be any particular request that triggers this > behavior. We have checked the server load when we discover the issue is > occurring and everything seems normal. The problem process does not seem to > be using any more memory or CPU than any of the other processes, and the > number of processes is fine. > > > > Has anyone experienced anything like this? Any ideas how we can track down > the problem? > > > > Thank you in advance for any help you can give us. > > > > Sharon
Re: Mixed Children on amd64
On Mon, Sep 20, 2010 at 7:00 PM, Vincent Veyron wrote: > When I use this new machine (it is currently disabled), my registered > users on occasion will retrieve the data from the demo account instead > of their own, as if the children spawned by the two apache2 processes > got mixed up. Are these just two virtual servers running from the same httpd.conf and server binaries? How does each one know which database to use? It seems unlikely that this is related to your processor change. - Perrin
Re: Mixed Children on amd64
Perrin Harkins wrote: On Mon, Sep 20, 2010 at 7:00 PM, Vincent Veyron wrote: When I use this new machine (it is currently disabled), my registered users on occasion will retrieve the data from the demo account instead of their own, as if the children spawned by the two apache2 processes got mixed up. Are these just two virtual servers running from the same httpd.conf and server binaries? How does each one know which database to use? It seems unlikely that this is related to your processor change. I would be more nuanced. Sometimes, a problem in the logic, which was there all the time but never caused any actual problem because it was very unlikely, can become acute because of an apparently unrelated change. For example, events that almost never occurred "simultaneously" on a slower processor or with less memory, now start to happen "simultaneously" because the processor is much faster, or because there is more memory available to start more processes at the same time. And never mind the fact that a modern 64-bit processor is likely to have several cores, allowing it to actually run more than one process at the same time. So something which before never happened, or happened only once or twice a year, and was not reproducible and thus was ignored as a "cosmic ray hitting the CPU", now happens 2 or 3 times a day and cannot be ignored anymore. The original description of the problem really makes me think of something like that. I would bet my shirt that if you considered 2 instances of this application side by side, and imagined that they do each step in parallel, there would be a point where you would notice that something could go wrong. And with a faster CPU, things that can go wrong, do go wrong faster.
Re: Mixed Children on amd64
Vincent Veyron wrote: Hi, I wrote a web application for case management (legal, insurance) with mod_perl. The app is visible at : http://marica.fr/ I recently purchased a 'dedibox' (http://www.online.net/serveur-dedie/offre-dedibox-v3.xhtml) intended to host it, and configured it with Debian. My setup is as follow : Two instances of Apache2 sit on the server, one with mod_ssl enabled, the other without. The former serves registered users on port 443, the latter serves only the demo account which is visible on the home page. Hi again. First again a couple of personal opinions : 1) I do not think that anything that you did in terms of compiling Apache, could be the origin of the problem which you describe. 2) It is extremely unlikely (I would say practically impossible) that two "Apache children", if they are different processes, would get mixed up. The whole architecture of CPUs and OS'es makes that very unlikely. You would have much bigger problems otherwise. But a couple of things puzzle me in your setup. 1) if you are using Linux Debian, they why do you go through the trouble of compiling Apache etc.. ? The Debian packagers have done a good job, and you can just install perl, Apache and mod_perl with 3 commands : apt-get install perl apt-get install apache2 apt-get install libapache2-mod-perl2 The above takes about 2 minutes, and I do this on all our production servers, without any problems until now. It also makes it much easier to keep the server up-to-date in terms of security updates etc.. 2) if you are running only 2 sites, of which only one is HTTPS, you could run it all in one single Apache instance. It is no problem to run a single VirtualHost as a HTTPS host on its own port 443, and other multiple HTTP VirtualHost's on port 80. The problem is only when you want to run several HTTPS hosts. I did try the first page of the demo site. I am using Firefox (with a HttpFox plugin, usually very handly, but I did not need it here). I can see that there was a cookie set, which in Firefox shows up as : Host: marica.fr name: session path: / secure: for any type of session (so it is not marked secure) I don't know of course how the cookie is set for the secure part of the site. But if the parameters above are set the same way, then there would be a least a potential for cookies to get mixed-up between the open and the secure part of the site. The "https" versus "http", or the port (443 or 80), are /not/ part of that which, for a browser, makes a cookie unique. I do not know exactly how Apache::Session and your cookie-based authentication work, but I suppose it is possible that some link somewhere in your application causes the users to switch between the secure and non-secure part of the site, and that if they already have a "session" cookie, the authentication mechanism would just accept it at face value and direct the call to the wrong place. Maybe one thing you could do, since these are two servers with a separate configuration, is at least to change the name of the cookie in one of them (for example, name it "secure-session" in the secure server). That would make them 2 separate cookies, and maybe avoid the confusion (or show the problem right away, by popping up a login page as soon as they click the "bad" link).
Re: Strange process behavior
Hi, I think I sent my last reply just to Tuomo, so am sending again to the list.. Thanks for the suggestion. I checked our error logs and did not find that message anywhere. We added some logging to several of our scripts to log when $r->args and $q->query_string do not match. What we find for the bad process is that $r always has the correct data for the new request, but the value of $q->query_string never changes once the problem starts. Here's a shortened example of what we are logging: [Sat Sep 18 01:14:06 2010] [error] INFO LOGGER: Mismatched query strings: r = item_code=WW&item_no=293845&view=details, q = sku=828752&slide=2||PID: 2309 [Sat Sep 18 01:14:10 2010] [error] INFO LOGGER: Mismatched query strings: r = sp=102686, q = sku=828752&slide=2||PID: 2309 [Sat Sep 18 01:14:12 2010] [error] INFO LOGGER: Mismatched query strings: r = , q = sku=828752&slide=2||PID: 2309 [Sat Sep 18 01:14:20 2010] [error] INFO LOGGER: Mismatched query strings: r = N=1108281+5401+10006, q = sku=828752&slide=2||PID: 2309 [Sat Sep 18 01:18:53 2010] [error] INFO LOGGER: Mismatched query strings: r = item_no=312108, q = sku=828752&slide=2||PID: 2309 This will continue until the process with PID 2309 dies, and it happens with any script that is requested. We are doing some internal redirects as Perrin mentioned, but we are not redirecting every time we see this happen, so I can't say that redirects are causing the problem. At the same time this is happening, all other child processes are behaving normally. Sharon On Tue, Sep 21, 2010 at 8:28 AM, Tuomo Salo wrote: > Are you getting "variable will not stay shared" messages in your error_log? > > If you are, you might be using a lexical variable outside of the function > it was defined in. When this happens, you'll get a closure, which is a > useful tool sometimes, but for us regular people it means that old data > will unexpectedly stay in the variables from one request to the other. > > The fix is to make sure that every variable (that is not meant to be > global) > is declared as "my" inside any sub where it is used. > > > http://modperlbook.org/html/22-2-1-Value-of-x-will-not-stay-shared-at-line-5.html > > -Tuomo > > On Mon, Sep 20, 2010 at 11:52:50AM -0400, Sharon Pattison wrote: > > Hi, > > > > > > We are experiencing a strange problem on our web servers for our site, > and > > are not sure if the problem is mod_perl related, but thought we would > turn > > here for help in case someone else has experienced this issue. > > Unfortunately, we are not sure exactly when the problem started, so we > can’t > > point to any particular change that might have prompted it. > > > > > > > > We are using Apache 1.3.42 with mod_perl 1.31 and Perl 5.8.3 on Solaris > 9. > > What is happening is that a particular Apache child process starts having > an > > issue where the wrong content is being returned for a request. The > content > > returned is always from the correct script, but it has the wrong data. > For > > example, if we request a product detail page for a particular product, we > > might get a different product, or we might get a message that the product > > does not exist, when it does. > > > > > > > > After adding a lot of logging to our logs, we discovered that the Apache > > request object will have the correct information, but the CGI object > never > > seems to get the global values reset, so is always has the query_string > of > > the request where things started to go wrong. The process will continue > to > > get and respond to requests, but the CGI object’s query_string never > > changes, so the user either gets no information or the wrong information. > > > > > > > > For example, if we had a request for mysite.com/product?sku=12345, and > the > > process that served this request starts to have the problem, then we see > > that for every request that process gets, logging $q->query_string gives > us > > ‘sku=12345’. It seems like the initialize_globals() CGI method is never > > being executed in the process cleanup. > > > > > > > > There doesn’t seem to be any particular request that triggers this > > behavior. We have checked the server load when we discover the issue is > > occurring and everything seems normal. The problem process does not seem > to > > be using any more memory or CPU than any of the other processes, and the > > number of processes is fine. > > > > > > > > Has anyone experienced anything like this? Any ideas how we can track > down > > the problem? > > > > > > > > Thank you in advance for any help you can give us. > > > > > > > > Sharon >
Re: Strange process behavior
On Tue, Sep 21, 2010 at 12:09 PM, Sharon Pattison wrote: > We added some logging to several of our scripts to log when $r->args and > $q->query_string do not match. What we find for the bad process is that $r > always has the correct data for the new request, but the value of > $q->query_string never changes once the problem starts. One quick thing to check: make sure you have the very latest version of CGI.pm installed! This often fixes mysterious CGI.pm problems. > Here's a shortened > example of what we are logging: I'd suggest you log enough that you can see all the requests sent to a process before it broke. Then try duplicating those with an LWP script and see if you can reproduce the breakage. - Perrin
Re: Mixed Children on amd64
Le mardi 21 septembre 2010 à 17:59 +0200, André Warnier a écrit :
a couple of things puzzle me in your setup.
>
> 1) if you are using Linux Debian, they why do you go through the trouble of
> compiling
> Apache etc.. ? The Debian packagers have done a good job, and you can just
> install perl,
> Apache and mod_perl with 3 commands :
> apt-get install perl
> apt-get install apache2
> apt-get install libapache2-mod-perl2
>
> The above takes about 2 minutes, and I do this on all our production servers,
> without any
> problems until now. It also makes it much easier to keep the server
> up-to-date in terms of
> security updates etc..
>
I found it more reliable for installation over various machines, it also
makes it easier for me to use different versions of
apache/perl/postgres. Mod_ssl complicates matters a bit too, so I have
better control that way. I seem to remember that performance is also
better, according to ApacheBench.
I realized after the fact that updates _are_ a problem. I suppose one
has to recompile from source. I'm hoping to script the process,
eventually.
> 2) if you are running only 2 sites
I do have a couple other virtual hosts on the server, which is the
reason for the http only server.
I also wanted to minimize the cpu load and separate the demo account
from the main database for security reasons, which is why that account
is served from the http server, with a different database connection.
> , of which only one is HTTPS, you could run it all in
> one single Apache instance. It is no problem to run a single VirtualHost as
> a HTTPS host
> on its own port 443, and other multiple HTTP VirtualHost's on port 80.
> The problem is only when you want to run several HTTPS hosts.
>
This sounds like what I'm doing now? You do need two httpd processes,
one that listens on port 80, the other on port 443.
>
> I did try the first page of the demo site. I am using Firefox (with a
> HttpFox plugin,
> usually very handly, but I did not need it here).
> I can see that there was a cookie set, which in Firefox shows up as :
> Host: marica.fr
> name: session
> path: /
> secure: for any type of session
> (so it is not marked secure)
>
> I don't know of course how the cookie is set for the secure part of the site.
> But if the parameters above are set the same way,
They are set the same way, both processes use the same modules.
> then there would be a least a potential
> for cookies to get mixed-up between the open and the secure part of the site.
> The "https" versus "http", or the port (443 or 80), are /not/ part of that
> which, for a
> browser, makes a cookie unique.
>
> I do not know exactly how Apache::Session and your cookie-based
> authentication work, but I
> suppose it is possible that some link somewhere in your application causes
> the users to
> switch between the secure and non-secure part of the site, and that if they
> already have a
> "session" cookie, the authentication mechanism would just accept it at face
> value and
> direct the call to the wrong place.
Save for a couple exceptions (style sheets and images), I enforce
relative links in my code, so I doubt a switch happens.
In any case, demo and registered users can log in both ways, http or
https; this does not change the logic of the database connection (see my
other reply for the code).
Their cookie is unique in my session database, and that session is tied
to their account, so they shouldn't see the demo user's data, regardless
of whether they are using https or not.
See the relevant code in the HeaderParser below :
=
sub handler {
my $r = shift ;
#on suppose que la session n'est pas valide, jusqu'à preuve du contraire
my $redirect_to_login = 1 ;
my $j = Apache2::Cookie::Jar->new($r) ;
#get cookie from request headers
my $cookie = $j->cookies('session') ;
if (defined $cookie) {
#cookie string : session=151b733d5b92679da9177ad9c9dd7d7c
my ($key,$session_id) = split (/=/,$cookie) ;
my %session=() ;
#ne continuer que si on a une clé de session
if ( $session_id ) {
#create session
eval {
tie %session, 'Apache::Session::Postgres', $session_id, {
DataSource => 'dbi:Pg:dbname=marica_sessions',
UserName => $r->dir_config('db_user'),
Password => undef,
Commit => 1
}
} ;
#erreur avec la clé de session
if ($@) {
if ($@ =~ /^Object does not exist in the data store/) {
#session supprimée, rediriger vers login
} else {
warn $@ ;
return Apache2::Const::SERVER_ERROR
}
} else { #on a une session; valider son time_to_live
#vérifier si la session n'a pas expiré
if ( $session{time_to_live} >= time() ) {
[...]
=
> Maybe one thing you could do, since these are tw
Re: Mixed Children on amd64
Hi,
Replying here to both of you for convenience
Le mardi 21 septembre 2010 à 16:51 +0200, André Warnier a écrit :
> Perrin Harkins wrote:
> >
> > Are these just two virtual servers running from the same httpd.conf
> > and server binaries? How does each one know which database to use?
Each server has its own httpd.conf and binaries, compiled in different
directories. Both processes use the same perl modules.
After login, a headerParser validates the session; demo users have a
'demo_' prefix appended to their name, and in that case the cached
database connection used is the demo one.
Below is the relevant code in the headerparser:
==
#list of database names (demo, main)
my @databases = $r->dir_config->get('db_name');
#identity of demo user
my $demo_username = $r->dir_config('demo_username');
#demo user connects to demo db
my ($db_type,
$db_name,
$db_user,
$db_pwd,
%attributes ) =
( $r->dir_config('db_type'),
( $r->pnotes('session')->{username} =~ /$demo_username/ ) ?
$databases[1] : $databases[0],
$r->dir_config('db_user'),
$r->dir_config('db_pwd'),
$r->dir_config->get('attributes') );
eval {
$dbh = DBI->connect_cached( "DBI:$db_type:dbname=$db_name",
$db_user, $db_pwd, \%attributes );
}
=
> Sometimes, a problem in the logic, which was there all the time but never
> caused any
> actual problem because it was very unlikely, can become acute because of an
> apparently
> unrelated change.
> For example, events that almost never occurred "simultaneously" on a slower
> processor or
> with less memory, now start to happen "simultaneously" because the processor
> is much
> faster, or because there is more memory available to start more processes at
> the same time.
> And never mind the fact that a modern 64-bit processor is likely to have
> several cores,
> allowing it to actually run more than one process at the same time.
> So something which before never happened, or happened only once or twice a
> year, and was
> not reproducible and thus was ignored as a "cosmic ray hitting the CPU", now
> happens 2 or
> 3 times a day and cannot be ignored anymore.
> The original description of the problem really makes me think of something
> like that.
> I would bet my shirt that if you considered 2 instances of this application
> side by side,
> and imagined that they do each step in parallel, there would be a point where
> you would
> notice that something could go wrong.
> And with a faster CPU, things that can go wrong, do go wrong faster.
>
Indeed, this machine has more memory and a faster processor than the
previous ones, but I have very low traffic. Below is the result of `top`
(the machine was facing the internet until yesterday). As you can see,
it's not stressed by any means.
=
top - 21:19:18 up 35 days, 13:24, 1 user, load average: 0.00, 0.00,
0.00
Tasks: 87 total, 1 running, 86 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 2028176k total, 1544980k used, 483196k free, 274400k buffers
Swap: 1044216k total,0k used, 1044216k free, 1031144k cached
=
The previous machines had similar statistics, but users noticed the
problem as soon as I put this one up.
Re: Mixed Children on amd64
Vincent Veyron wrote: ... , of which only one is HTTPS, you could run it all in one single Apache instance. It is no problem to run a single VirtualHost as a HTTPS host on its own port 443, and other multiple HTTP VirtualHost's on port 80. The problem is only when you want to run several HTTPS hosts. This sounds like what I'm doing now? You do need two httpd processes, one that listens on port 80, the other on port 443. No you don't. If in one Apache you say Listen 80 Listen 443 and it will listen to both ports. And then you can say NameVirtualHost *:80 ServerName A .. ServerName B .. ... NameVirtualHost *:443 ServerName C .. (but only once, for HTTPS; the reason for that is longer to explain). ... Sorry, really analysing the code is a bit beyond my commitment. I am just trying to give you ideas of what to look for. And to discourage you from looking in the wrong direction, because the idea that 2 Apache processes could be mixing their data sounds really far-fetched to me. Maybe one thing you could do, since these are two servers with a separate configuration, is at least to change the name of the cookie in one of them (for example, name it "secure-session" in the secure server). That would make them 2 separate cookies, and maybe avoid the confusion (or show the problem right away, by popping up a login page as soon as they click the "bad" link). Even supposing a bad link exists, the browser always sends the same cookie, regardless of whether it's using http or https. Yes, and that is what I mean. Whether users stray through the secure or non-secure site, there is only ever one cookie. And if it is not marked secure, the browser will send that same cookie, no matter which site the users link to. And the server receiving the cookie, at least in the authentication part of the code, will not see the diference, and will let them in as long as for the session referenced in the cookie, there is a valid record in the database. So IF users would go from one server to the other, you would probably never know, because they will not be stopped from doing that. And that could certainly be a good reason why some users see demo data some time, instead of theirs. I am not saying that it /is/ the problem. What I am saying is that if you had a different cookie name for each site (which should be easy to do), then for sure the above could not happen, and you could eliminate one area from your search. Be humble. On one side, there is Apache code, which is extensively tested and running on hundreds of thousands of sites. On the other side, there is your code, which runs on just a few sites. If there is a problem somewhere, where is it most likely to be ?
Help: unwanted extra characters in output
Hi - I have a problem which I can't seem to solve. I am busy implementing a webdav solution in mod_perl and as you may know, the output is XML based. The problem is that the output generated by mod_perl contains additional characters before and after the XML which the webdav clients don't seem to like. If I convert this script to a CGI, it works 100% and the extra characters is not present. Any ideas? Here is the example trace (check the "20d" and "0" before and after the XML): PROPFIND /webdav/ HTTP/1.1 Host: test:81 User-Agent: cadaver/0.22.3 neon/0.25.5 Connection: TE TE: trailers Depth: 0 Content-Length: 297 Content-Type: application/xml Authorization: Basic dGVzdHVzZXIxOnBhc3N3b3Jk http://apache.org/dav/props/"/> HTTP/1.1 200 OK Date: Wed, 22 Sep 2010 04:25:57 GMT Server: Apache/2.2.3 (CentOS) Cache-Control: max-age=0 Expires: Wed, 22 Sep 2010 04:25:57 GMT Connection: close Transfer-Encoding: chunked Content-Type: text/xtml; charset="utf-8" 20d http://apache.org/dav/props/";> 2010-09-14T05:09:29Z F 4096 HTTP/1.1 200 OK 0
