I greatly appreciate everyone's input to this problem. Due to workload, I have not yet had an opportunity to follow up with all the suggestions, but I plan to if I am not able to resolve this problem by other means.
I do have a follow-up question about another possible cause of the problem. I realized tonight after looking over some of my @INC bootstrapping code that for some time the code was actually use(ing) MyApp *before* the "require MyApp" line in myapp_server.pl. Obviously, an unnecessary step, but I started wondering if it might be the cause of the problem, especially after I looked back over my SVN history and determined that the "use" call went into production about 10 days before our 1st report of a crossed session. Although, thinking back to when the first report came through our production engine was FastCGI, which doesn't appear to have ever had the 2 require(use) calls. I know that CORE::require isn't *supposed* to load a module if it's already loaded *and* that Catalyst->setup does a check of $class->setup_finished to ensure that ->setup isn't called twice, but I thought I would ask anyway. As an added note I have never once seen the "Running setup twice is not a good idea." warning in my logs. Thanks again, Matt Pitts > -----Original Message----- > From: Matt Pitts [mailto:[email protected]] > Sent: Thursday, January 08, 2009 10:20 AM > To: The elegant MVC web framework > Subject: [Catalyst] Production session issue - commercial support > inquiry? > > A Cat app for one of our clients has been experiencing random session > cross-over issues for the past several months. The first reported > instance was on Oct. 16 and I have spent countless hours looking at any > changes that took place around that time to try to isolate the problem > without success. > > Here is an overview of the application setup: > > - Catalyst 5.70014 > - Catalyst::Plugin::Session 0.20 > - single Apache 2.2 load balancer front end handling remote SSL and > non-SSL requests > - two backend servers running Cat app via HTTP::Prefork for non-SSL > traffic > - each backend also runs Apache/mod_ssl + mod_fastcgi in order to > provide an SSL channel to the backends > - session storage via DBIC (was originally memcached) > > The symptoms of the session issues: > > - users "seeing" other user's items in their Cart > - "club" members updating their information, but the update gets tied > to another member > > I have personally experienced the issue myself while the site was under > a bit of traffic - I added an item to my Cart and I had other items in > there that I didn't touch. I grabbed my cookie and tracked down my > session and it was pointing to a cart_id that had all the items in it > that were showing on the page. The only conclusion I can draw from this > is that two separate sessions somehow got mapped to the same cart_id - > which I don't know where this could be happening at. I'm fairly > confident that it's not my Cart code because the issue is also > happening > on another part of the site - a member's area. > > In an effort to track down the issue I have done the following (not > necessary in chronological order): > > - put checks in place during Cart retrieval to help ensure a good > "load" > Result: no change > > - applied both "finalize_race_condition" and "flash_in_stash" patches > from Sergio Salvi > > (http://dev.catalyst.perl.org/svnweb/Catalyst/browse/branches/Catalyst- > P > lugin-Session/both) > Result: no change > > - taken one of the backends out of the pool for several days > Result: no change > > - switched session storage to FastMmap while using only a single > backend > Result: no change > > - switched session storage to DBIC > Result: no change > > - updated C::P::Session from 0.19 to 0.20 > Result: no change > > - changed deployment model from FastCGI to HTTP::Prefork and back > Result: no change > > - setup SHH tunnel for SSL traffic to go directly to HTTP::Prefork > backend instances (completely eliminate FastCGI components) > Result: no change > > - attempted to replicate problem in a test environment using a forking > site crawler I hacked up > Result: no success > > All in all, I'm fairly confident that this issue is either an error in > my code/setup of Catalyst and C::P::Session or it's a bug somewhere in > Catalyst or C::P::Session. I know there's been some traffic on the list > about session issues and some talk of refactoring the session code, but > I haven't seen anything that is relevant to this problem. > > Any obvious help or suggestions are *greatly* appreciated, but... > > I'm to the point that I need additional help with this problem and > since > I'm the only Linux/Perl guys in our shop I'm calling out. My company is > committed to getting this problem resolved and we want to get an idea > of > what commercial support is available. Our preference would be on-site > support - we are located in Greensboro, NC, USA - but we'll we can work > with remote support if needed. If you're currently providing commercial > support specifically for Catalyst applications and you're interested in > the work, please feel free to contact me privately with your > information > and rates. > > Much thanks and appreciation, > > Matthew Pitts > Senior Engineer > Software and Linux Solutions > A3 IT Solutions, LLC > [c] 336.202.3913 > [o] 336.389.1101 ex117 > [e] [email protected] > > > > _______________________________________________ > List: [email protected] > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: http://www.mail- > archive.com/[email protected]/ > Dev site: http://dev.catalyst.perl.org/ _______________________________________________ List: [email protected] Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[email protected]/ Dev site: http://dev.catalyst.perl.org/
