We had another major incident of session cross-over this morning which happened right after I pushed out a small change and reload/restarted the instances.
I personally experienced the session cross-over twice. In each instance I captured my session id and went to compare it to a dump of all sessions and found that no other session had the same data as mine. However, the 2 session id's *were* the same and I had cleared my cookies between these two instances. So, I could be getting dups out of generate_session_id, but I'm also wondering if it's not some weird process issue where the same session ID is not being generated twice, but is just getting shoved into the response headers of 2 different requests. Anyone know if this is plausible? If P::Session is generating duplicate IDs I doubt I'd be the only one having the issue. BTW, the cat app only averages about 12 req/s and this issue doesn't seem to be very dependent on traffic. Along the lines of the process question... What, if any, are the ramifications of running two instances of the app (one HTTP::Prefork and one FCGI) out of the same "root". In my particular case the FCGI instance is for talking to a local Apache instance that is providing SSL access and the HTTP::Prefork instance is to provide direct HTTP access to areas where SSL isn't needed. I wasn't originally worried about this because they're separate process, but right now I'm looking at anything that might be causing my problems. Could a situation like this cause strange behavior in regards to session ids. Right now I have a set of changes primed to push out that completely removed P::Session and uses P::CookiedSession instead. I guess if I don't get a resolution to the issues with P::Session by Monday I'll be pushing this out and then I'll know it's not P::Session problem if it continues. TIA -matt > -----Original Message----- > From: Matt Pitts [mailto:[email protected]] > Sent: Thursday, January 08, 2009 10:20 AM > To: The elegant MVC web framework > Subject: [Catalyst] Production session issue - commercial support > inquiry? > > A Cat app for one of our clients has been experiencing random session > cross-over issues for the past several months. The first reported > instance was on Oct. 16 and I have spent countless hours looking at any > changes that took place around that time to try to isolate the problem > without success. > > Here is an overview of the application setup: > > - Catalyst 5.70014 > - Catalyst::Plugin::Session 0.20 > - single Apache 2.2 load balancer front end handling remote SSL and > non-SSL requests > - two backend servers running Cat app via HTTP::Prefork for non-SSL > traffic > - each backend also runs Apache/mod_ssl + mod_fastcgi in order to > provide an SSL channel to the backends > - session storage via DBIC (was originally memcached) > > The symptoms of the session issues: > > - users "seeing" other user's items in their Cart > - "club" members updating their information, but the update gets tied > to another member > > I have personally experienced the issue myself while the site was under > a bit of traffic - I added an item to my Cart and I had other items in > there that I didn't touch. I grabbed my cookie and tracked down my > session and it was pointing to a cart_id that had all the items in it > that were showing on the page. The only conclusion I can draw from this > is that two separate sessions somehow got mapped to the same cart_id - > which I don't know where this could be happening at. I'm fairly > confident that it's not my Cart code because the issue is also > happening > on another part of the site - a member's area. > > In an effort to track down the issue I have done the following (not > necessary in chronological order): > > - put checks in place during Cart retrieval to help ensure a good > "load" > Result: no change > > - applied both "finalize_race_condition" and "flash_in_stash" patches > from Sergio Salvi > > (http://dev.catalyst.perl.org/svnweb/Catalyst/browse/branches/Catalyst- > P > lugin-Session/both) > Result: no change > > - taken one of the backends out of the pool for several days > Result: no change > > - switched session storage to FastMmap while using only a single > backend > Result: no change > > - switched session storage to DBIC > Result: no change > > - updated C::P::Session from 0.19 to 0.20 > Result: no change > > - changed deployment model from FastCGI to HTTP::Prefork and back > Result: no change > > - setup SHH tunnel for SSL traffic to go directly to HTTP::Prefork > backend instances (completely eliminate FastCGI components) > Result: no change > > - attempted to replicate problem in a test environment using a forking > site crawler I hacked up > Result: no success > > All in all, I'm fairly confident that this issue is either an error in > my code/setup of Catalyst and C::P::Session or it's a bug somewhere in > Catalyst or C::P::Session. I know there's been some traffic on the list > about session issues and some talk of refactoring the session code, but > I haven't seen anything that is relevant to this problem. > > Any obvious help or suggestions are *greatly* appreciated, but... > > I'm to the point that I need additional help with this problem and > since > I'm the only Linux/Perl guys in our shop I'm calling out. My company is > committed to getting this problem resolved and we want to get an idea > of > what commercial support is available. Our preference would be on-site > support - we are located in Greensboro, NC, USA - but we'll we can work > with remote support if needed. If you're currently providing commercial > support specifically for Catalyst applications and you're interested in > the work, please feel free to contact me privately with your > information > and rates. > > Much thanks and appreciation, > > Matthew Pitts > Senior Engineer > Software and Linux Solutions > A3 IT Solutions, LLC > [c] 336.202.3913 > [o] 336.389.1101 ex117 > [e] [email protected] > > > > _______________________________________________ > List: [email protected] > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: http://www.mail- > archive.com/[email protected]/ > Dev site: http://dev.catalyst.perl.org/ _______________________________________________ List: [email protected] Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[email protected]/ Dev site: http://dev.catalyst.perl.org/
