requests possibly not reaching the log phase
(apologies if this is a dupe, I originally sent from the wrong address) Hi there, I have an unusual problem - a large e-commerce site integrated with Authorize.net for card payments which appears to be failing to log some requests. The Authorize.net system makes HTTP POST requests to our server, and about 1 in every 500 transactions, the Authorize.net system reports a timeout and there's no trace of the request in our logs. Authorize.net won't investigate in any detail because their system is reporting that the request simply timed out. I'm using mod_perl but not hooking into the logging phase, and using a mod_log CustomLog directive which outputs the usual stuff, plus request time, connection state, and PID. So for now, I'm assuming a request is being sent but the log handler phase isn't running. The only way I can make this happen in a test environment is by opening a TCP connection and then closing it without sending any data. Are there any other reasons the log phase wouldn't be run? My next steps are to add simple request logging during the Trans phase, and failing that, packet sniffing, but this is a live high-traffic server so I'm trying to avoid that if possible. cheers John
Re: requests possibly not reaching the log phase
John ORourke wrote: I have an unusual problem - a large e-commerce site integrated with Authorize.net for card payments which appears to be failing to log some requests. The Authorize.net system makes HTTP POST requests to our server, and about 1 in every 500 transactions, the Authorize.net system reports a timeout and there's no trace of the request in our logs. Authorize.net won't investigate in any detail because their system is reporting that the request simply timed out. If httpd isn't logging anything, the most likely explanation is that the request isn't reaching httpd at all. How reliable is your network between their system and yours? Are there any load balancing devices or other network magic in the way that could potentially be misconfigured? My next steps are to add simple request logging during the Trans phase, and failing that, packet sniffing, but this is a live high-traffic server so I'm trying to avoid that if possible. Packet sniffing will answer the question was there any evidence of a request, and would probably be the least invasive way of measuring this. It's always useful to see what the network is actually doing, rather than what you think it's doing. Regards, Graham --
Re: requests possibly not reaching the log phase
Graham Leggett wrote: If httpd isn't logging anything, the most likely explanation is that the request isn't reaching httpd at all. How reliable is your network between their system and yours? Are there any load balancing devices or other network magic in the way that could potentially be misconfigured? That gives me a few new places to hunt down the issue, thanks. Packet sniffing will answer the question was there any evidence of a request, and would probably be the least invasive way of measuring this. It's always useful to see what the network is actually doing, rather than what you think it's doing. Definitely - I've decided to bite the bullet and run tshark in the background, hopefully that will turn up some new information. Debugging an error which happens at random a few times a month is an interesting challenge! cheers John
Re: [mod_fcgid] Feedback / Suggestions
Jeff Trawick wrote: On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank ef-li...@email.de wrote: In the interim, is mod_fastcgi really that bad? mod_fastcgi is fine for handling GET/POST requests, but it fails to implement Authorization or Authenication. So yes mod_fastcgi is really bad. mod_fcgid is a very welcome as a supported httpd module. Barry
Re: requests possibly not reaching the log phase
You might also take a look into global apache's error log. There are very rare cases when PHP (or, probably, other modules) causes segmentation faults in apache's child processes which prevents any request info to be written to access log. The last time I had problems with this issue it was caused by Oracle PHP extension (oci). 2009/12/3 John ORourke john-p...@o-rourke.org That gives me a few new places to hunt down the issue, thanks. Regards, Paul
Re: requests possibly not reaching the log phase
On 3 Dec 2009, at 08:43, John ORourke wrote: So for now, I'm assuming a request is being sent but the log handler phase isn't running. Seems improbable, though mod_perl adds an extra layer. Graham's suggestion looks more plausible. My next steps are to add simple request logging during the Trans phase, You can do that by adding a match-and-do-nothing RewriteRule and setting RewriteLogLevel to log it. Or the third-party mod_security will log a great deal more for you. btw, this should be on the users@ list. -- Nick Kew
Re: requests possibly not reaching the log phase
John ORourke john-p...@o-rourke.org writes: The Authorize.net system makes HTTP POST requests to our server, and about 1 in every 500 transactions, the Authorize.net system reports a timeout and there's no trace of the request in our logs. Authorize.net won't investigate in any detail because their system is reporting that the request simply timed out. This could happen in Apache 2 if the web server timed out before reading the complete request. (Apache 1.3 logs a 408; Apache 2 doesn't; Apache trunk will log a 4xx of some sort depending on how much of the request was read.) If that's what's happening, it would indicate a problem outside of Apache. You could confirm it by backporting the fix from https://issues.apache.org/bugzilla/show_bug.cgi?id=39785 so logging would occur. Then you might need to resort to packet tracing or something like that to figure out exactly what's going on. Dan
Re: [mod_fcgid] Feedback / Suggestions
On Thu, Dec 3, 2009 at 5:57 AM, Barry Scott barry.sc...@onelan.co.uk wrote: Jeff Trawick wrote: On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank ef-li...@email.de wrote: In the interim, is mod_fastcgi really that bad? mod_fastcgi is fine for handling GET/POST requests, but it fails to implement Authorization or Authenication. So yes mod_fastcgi is really bad. mod_fastcgi has supported this for many years: http://www.fastcgi.com/drupal/node/25#FastCgiAuthorizer http://www.fastcgi.com/drupal/node/25#FastCgiAuthenticator -- Eric Covener cove...@gmail.com
Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init
Graham Leggett wrote: On a Redhat machine (Fedora/RHEL/Centos), search /etc/rc.d/init.d/functions for functions called daemon, status and killproc. These functions provide similar but incompatible functionality to that provided by apachectl, and only exist on Redhat derived machines. Ok, so they want to roll their own. Sounds like a maintainer issue. What does this say for using our httpd rpm for an Ubuntu or other distribution of linux? The startup script is far too trivial to justify jumping through hoops to try and make apachectl work like Redhat's init. It's caused us enough grief already, thus the fix. IMHO, if it is fundamentally incompatible with apachectl and non-redhat distributions, let the the packagers tweak and take the zany customizations out from under our problem set.
Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init
Gregg L. Smith wrote: Original Message --- Finally, I have yet to see any feedback on the pcre mandatory dependency issue. Comments? Personally, I thought your Monopoly metaphor was quite on target. libz, openssl, lua = batteries not included apr, apu, pcre = drive train not included. And what is passing for an excuse for a local PCRE install these days probably doesn't look like 7.8 or later, with various fixes we are vulnerable to. This does not leave me with a warm and fuzzy feeling. As a user, is the pcre 8.0 I've built going to expose me to risks that your maintained 7.8 does not? If yes, then I'd prefer your maintained one. After all, who knows better than you what will interact with your code to produce problems. Regardless of merit, who will ultimately get blamed in the end? Could your reputation be tarnished? Can you completely divorce yourself from something your software requires to run? I'm referring to pre v7 chaos. And mostly not referring to modern linux distros. The 'Jump Ship' factor; To me, and I'm probably wrong, it sounds like Mr. Felt's comment was an ultimatum of sorts as 'indefinitely' is a pretty strong word. With this issue you have created a deal with it or jump ship ultimatum which could very well leave some people scrambling to get off. Each person is going to inevitably weigh the pain factor, the pain of dealing with it over the pain of jumping ship. I consider myself lucky that my second attempt to deal with it was successful, or so it seems so far anyway, but I never know from day to day. Agreed that ease-of-adoption is going to be the usual, first barrier to anyone jumping aboard 2.4 from 2.2, 2.0, or even still from 1.3. I may be wrong but as an outsider looking in, I see you wanting to stop maintaining/including the gear box and are instead spending the time on adding more optional gadgets to choose from (some of the third party modules you've taken over). In the end, I'd prefer having a reverse gear over the rear window defogger. You are also loosing all control of a required piece of equipment, this has got to make some of you at least cringe a little. I'm not 100% sure I understand what you are saying here. Drop the gearbox and let them assemble their own transmission? Or distribute a most modern transmission that the user can ignore or swap out if they want to install their own? Sorry for the outburst, but you opened the door for, and I've said what I've wanted to for some time now, thanks for listening. Corrections and daggers welcomed. No problems, thanks for chiming in.
[RESULTS] [VOTE] Release httpd 2.3.4-alpha
Vote Results: +1 (binding): Sander Temme, Paul Querna, Joe Orton, Niklas Edmundsson, +1: Gregg Smith +/-0: Rainer Jung -1: William A. Rowe, Jr. Vote passes. I'll push out the tarballs to start getting mirrors, and hack on an announcement email for tomorrow. On Wed, Nov 25, 2009 at 2:43 PM, Paul Querna p...@querna.org wrote: Test tarballs for Apache httpd 2.3.4-alpha are available at: http://httpd.apache.org/dev/dist/ Your votes please; +/- 1 [ ] Release httpd-2.3.4 as Alpha Vote closes at 18:00 UTC on Monday November 30 2009. May your Thanksgiving be filled with Turkey and httpd testing, Thanks, Paul
Re: [RESULTS] [VOTE] Release httpd 2.3.4-alpha
Paul Querna wrote: Vote Results: +1 (binding): Sander Temme, Paul Querna, Joe Orton, Niklas Edmundsson, +1: Gregg Smith +/-0: Rainer Jung -1: William A. Rowe, Jr. Vote passes. I'm sorry. I explicitly insisted on a vote on the -deps package seperately from the 2.3.4 package, because it was entirely reasonable that Sander Temme, Paul Querna, Joe Orton, Niklas Edmundsson, or Gregg Smith reviewed -only- the httpd-2.3.4-alpha.tar.xx package alone. Gentlemen, if you could verify that your vote was for *both* the core package and the -deps.tar.xx package, that might keep Paul on his schedule (or fall against the proposed -deps.tar.xx package). Remember your -deps vote is to approve the release of apr 1.4.0-dev and the apr-util 1.4.0 dev, and the API versioning rules will bind from that release forwards.
Re: [VOTE] Release httpd 2.3.4-alpha
Paul Querna wrote: I don't agree that we can't release a bundled unreleased version of APR, we did this for many versions of httpd 2.0.x and 2.1.x. It definitely isn't preferred, but that's the APR project's problem. Look, your argument simply doesn't fly. In httpd 2.0 timeframe we were only shipping apr-0.9.x - it did NOT have the same API/ABI constraints (some of them, but not all). All of those intermediary releases kept the ABI rules of APR. Now that you have shipped immediately while ignoring my objection, I'll treat all +1's as binding no matter if they approved both of the pieces or not, and have tagged 1.4.0 of both apr and apr-util. We have no alternative, or else all author's VERSION_MAJOR/MINOR tests are invalid. It becomes up to the APR project if this aught to be 1.4.0 or burn a number and move on. For 1.4 initial release, I want to pick up Branko's fix, so I plan to label this 1.4.1. No intention of tagging apr-util yet till we decide if it can be API frozen, so if we dislike the current includes/ tree, it will end up being deprecated interfaces and version 1.5.0 already.
Re: [RESULTS] [VOTE] Release httpd 2.3.4-alpha
On Thu, Dec 3, 2009 at 6:21 PM, William A. Rowe Jr. wr...@rowe-clan.net wrote: Remember your -deps vote is to approve the release of apr 1.4.0-dev and the apr-util 1.4.0 dev, and the API versioning rules will bind from that release forwards. The APR versioning rules are hopelessly broken if a tarball snapshot of the 1.4.x branch before a GA release casts the API in stone. Surely I misunderstood you.
Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init
William A. Rowe Jr. wrote: Ok, so they want to roll their own. Sounds like a maintainer issue. What does this say for using our httpd rpm for an Ubuntu or other distribution of linux? Ubuntu is Debian based, and uses the .deb packaging format, and startup scripts derived from the Debian layout. If someone was to donate debian packaging for httpd, I would expect one or two files to appear below build/deb, and that would be all that would be needed. IMHO, if it is fundamentally incompatible with apachectl and non-redhat distributions, let the the packagers tweak and take the zany customizations out from under our problem set. Apachectl is archaic and largely broken for most people - it made sense ten years ago, it makes a lot less sense today. The pattern followed by most modern unix based packaging is for an application to drop a snippet of config contained in a discrete file in a directory ending in .d. So you have /etc/httpd/conf.d/snippet.conf, instead of a manual edit to /etc/httpd/conf/httpd.conf, and your httpd startup goes within an optional script called /etc/sysconfig/httpd instead of in a script file in a bin directory as apachectl wants. I understand Debian has different naming conventions, but otherwise the underlying principles are the same. In our case, we package up config files within standalone RPMs all of their own (suitably tagged and versioned), or we generate the config file using puppet. Editing a file in place is always painful and error prone, it is far less painful to provide a discrete file that can be dropped in and removed cleanly. Apachectl doesn't give us this - you need to edit apachectl directly to modify the command line parameters passed to httpd. As for the packagers tweaking and making zany customisations, that is exactly what they do now. For us it makes the their packaging unsuitable for our needs, simply because we tweak and make zany customisations for needs of our own. It is far less painful to take a vanilla RPM published by the ASF, and then tweak it for our needs, than it is to take a Fedora RPM, untweak all their customisations, and then retweak it with ours. Regards, Graham --
Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init
William A. Rowe Jr. wrote: I may be wrong but as an outsider looking in, I see you wanting to stop maintaining/including the gear box and are instead spending the time on adding more optional gadgets to choose from (some of the third party modules you've taken over). In the end, I'd prefer having a reverse gear over the rear window defogger. You are also loosing all control of a required piece of equipment, this has got to make some of you at least cringe a little. I'm not 100% sure I understand what you are saying here. Drop the gearbox and let them assemble their own transmission? Or distribute a most modern transmission that the user can ignore or swap out if they want to install their own? Bill, The latter in your statement, distribute the most modern transmission that the user can ignore or if they wish or swap out with something they prefer. Basically it's easy. If it must be there to build the darn thing, pass GO and collect the $$$, then it should be there. If not, then it becomes an option and that is left up to the user/builder ergo openssl zlib and lua. so +1 to including pcre in -deps (or srclib/pcre), basically as it is now in the other branches. In reality I probably shouldn't say anything anymore as I adapted and overcame, begrudgingly. It sure was a sore subject back between 2.3.1 and 2.3.2 and I guess it just held on and I could not resist making noise. Sorry this got into the wrong thread, I blame my webmail :) Gregg
Re: [RESULTS] [VOTE] Release httpd 2.3.4-alpha
Jeff Trawick wrote: On Thu, Dec 3, 2009 at 6:21 PM, William A. Rowe Jr. wr...@rowe-clan.net wrote: Remember your -deps vote is to approve the release of apr 1.4.0-dev and the apr-util 1.4.0 dev, and the API versioning rules will bind from that release forwards. The APR versioning rules are hopelessly broken if a tarball snapshot of the 1.4.x branch before a GA release casts the API in stone. Surely I misunderstood you. Is there a README indicating that the MAJOR/MINOR version tests for this particular tarball are not relevant/complete? No. This is not a snapshot. It is labeled httpd-2.3.4-alpha.tar.xx release. You surely don't misunderstand what I said. As for broken versioning rules, please take that to APR. Perhaps in retrospect, APR would consider an even/odds approach as httpd has for adding (even eliminating) interfaces during a development cycle.
Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init
Graham Leggett wrote: William A. Rowe Jr. wrote: Ok, so they want to roll their own. Sounds like a maintainer issue. What does this say for using our httpd rpm for an Ubuntu or other distribution of linux? Ubuntu is Debian based, and uses the .deb packaging format, and startup scripts derived from the Debian layout. The last I heard, the 'rpm' project is open source, free to be adopted by any platform. As for the rest of your comments, if we solve the general problem, I'm all for including it in the httpd tree. If we are solving specific packagers problems, I'm ok with placing this in the httpd.a.o domain, but we should move this nonsense outside of the development tree into a packaging tree.
