> -----Original Message-----
> From: Stuart Dallas [mailto:stu...@3ft9.com]
> Seriously? Errors like this should not be getting anywhere near your
> production servers. This is especially true if you're really getting 30k
> hits/s.

Don't get me started. I joined here almost a year ago. They didn't even have
a RCS or Wiki at the time. Nothing was OOP. There were no PHPDoc or even
comments in the code. They used to make each site by copying an existing one
and modifying it (ie. no shared libraries or resources). I could go on and
on. Suffice it to say we've made HUGE leaps and bounds (thanks to me!), but
there is only 3 of us developers here and no official test person let alone
a test team.

It is what it is. I'm doing the best I can with the limited resources
available to me.

And let me tell you a little secret, when you get to that scale, you see all
kinds of errors you don't see on your VM or even with a test team. DB
connections go away. Funny things happen to memcache. Concurrency issues
arise. Web bots and search engines rape, pillage and ravage your site in
ways that make you feel dirty. So yeah, you do hit weird situations and
cases you can't possibly test for, but show up in error logs.
> For a commercial, zero-hassle solution I can't recommend
> http://newrelic.com/ highly enough. Simple installation followed by highly
> detailed reports with zero issues (so far). They do a free trial of all
> pro features so you can see if it gets you what you need. And no, I don't
> work for them, I just think they've built a freakin' awesome product
> invaluable when diagnosing issues that only occur in production. I've
> used it on a site with that level of traffic, and I'm sure it won't be a
> problem, but you may want to only deploy it to a fraction of your
> infrastructure.

A quick look at that product seems interesting, but not what I really need.
We have a ton of monitoring solutions in place to get metrics and
performance data. I just need a good 'hook' to get details when errors

> If you want a homemade solution, the uncaught exceptions are easily dealt
> with... CATCH THEM, do something useful with them, and then die
> Rocket science this ain't!

Thanks captain obvious. :)

I can do that (and did do that), but again, at these scales, all the
text-book code you think you know starts to go out the window. Frameworks
break down. RDBMS topple over. You have to write things creatively, leanly
(and sometimes error on the side of 'assume something is there' rather than
'assume the worst' or your code spends too much time checking the edge
cases). Hit it and quit it! Get in. Get out. I can't put try/catch around
everything everywhere, it's just not efficient or practical. Even the SQL
queries we write are 'wide' and we pull in as much logical stuff as we can
in one DB call, get it into a memcache slab and then pull it out of there
over and over, rather than surgical queries to get small chunks of data
which would murder mySQL.

Part of the reason I took this job is exactly because of these challenges
and I've learned an incredible amount here (I've also had to wash the guilt
off of me some nights, as some code I've written goes against everything I
was taught and thought I knew for the past decade -- but it works and works
well -- it just FEELS wrong). We do a lot of things that would make my
college professors cringe. THAT is the difference between the REAL world and
the LAB. ;-)

> See the set_exception_handler function for an
> easy way to set up a global function to catch uncaught exceptions if you
> don't have a limited number of entry points.
> You can similarly catch the warnings using the set_error_handler function,
> tho be aware that this won't be triggered for fatal errors.

And this is the meat of the solution. Thanks! I'll look into these handlers
and see if I can inject it into someplace useful. I have high hopes for this

> But seriously... a minimal level of structured testing would prevent
> like this being deployed to your production servers. Sure, instrument to
> help resolve these issues now, but if I were you I'd be putting a lot of
> effort into improving your development process. Contact me off-list if
> like to talk about this in more detail.

See above. I have begged for even a single dedicated tester. I have offered
to sacrifice the open req I had for a junior developer to get a tester. That
resulted in them taking away the req because "clearly I didn't need the
developer then" and "we can just test it ourselves". You're preaching to the
choir my friend. I've been doing this for 15+ years at various companies.


PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Reply via email to