Hello, I've been trying to track down an intermittent problem with a Java web
application that is running on tcServer fronted by Apache HTTP Server. We
get intermittent "Server Unavailable / HTTP 500" errors, and when we do see
them, there is the same set of log statements written to the Apache HTTP Server
error log:
[Mon Jul 23 10:03:15 2012] [error] (70014)End of file found:
ajp_ilink_receive() can't receive header
[Mon Jul 23 10:03:15 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Jul 23 10:03:15 2012] [error] (120006)APR does not understand this error
code: proxy: dialog to 127.0.0.1:7071 (127.0.0.1) failed
We are not seeing any error messages in the tcServer logs.
I believe the issue is with the mod_proxy_ajp module but it's been very
difficult tracking down what exactly the problem is. What's interesting is
that this Apache / tcServer configuration is used with other applications that
work just fine and never have the intermittent 500 error. We also can run our
application strictly in Tomcat (no Apache front) without any intermittent
errors.
We haven't ruled out that there could be something in our Java application code
that is causing this, in combination with the mod_proxy_ajp module, but we have
hit a wall as to what this issue could be. Has anyone else experienced a
similar intermittent issue combined with the above error messages? Below is a
copy of the error log and some configuration settings.
Thanks, Tony
---------------------
Apache HTTP Error Log
---------------------
[Mon Jul 23 10:03:15 2012] [debug] mod_cache.c(141): Adding CACHE_SAVE filter
for /myapp/
[Mon Jul 23 10:03:15 2012] [debug] mod_cache.c(148): Adding CACHE_REMOVE_URL
filter for /myapp/
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_ajp.c(45): proxy: AJP:
canonicalising URL //127.0.0.1:7071/myapp/
[Mon Jul 23 10:03:15 2012] [debug] proxy_util.c(1506): [client ***cleansed***]
proxy: ajp: found worker ajp://127.0.0.1:7071/myapp for
ajp://127.0.0.1:7071/myapp/
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy.c(1020): Running scheme ajp
handler (attempt 0)
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_http.c(1963): proxy: HTTP:
declining URL ajp://127.0.0.1:7071/myapp/
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_ajp.c(681): proxy: AJP: serving
URL ajp://127.0.0.1:7071/myapp/
[Mon Jul 23 10:03:15 2012] [debug] proxy_util.c(2011): proxy: AJP: has acquired
connection for (127.0.0.1)
[Mon Jul 23 10:03:15 2012] [debug] proxy_util.c(2067): proxy: connecting
ajp://127.0.0.1:7071/myapp/ to 127.0.0.1:7071
[Mon Jul 23 10:03:15 2012] [debug] proxy_util.c(2193): proxy: connected /myapp/
to 127.0.0.1:7071
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[0] [x-forwarded-for] = [***cleansed***]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[1] [Host] = [***cleansed***]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[2] [Connection] = [keep-alive]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[3] [User-Agent] = [Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[4] [Accept] =
[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[5] [Accept-Encoding] = [gzip,deflate,sdch]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[6] [Accept-Language] = [en-US,en;q=0.8]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[7] [Accept-Charset] = [ISO-8859-1,utf-8;q=0.7,*;q=0.3]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[8] [Cookie] = [SSOTOKEN=***cleansed***]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[9] [SSO_LOGIN] = [***cleansed***]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[10] [SSO_ID] = [***cleansed***]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(290): ajp_marshal_into_msgb:
Header[11] [SSO_EMAIL] = [***cleansed***]
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(450): ajp_marshal_into_msgb:
Done
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_ajp.c(268): proxy:
APR_BUCKET_IS_EOS
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_ajp.c(273): proxy: data to read
(max 8186 at 4)
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_ajp.c(288): proxy: got 0 bytes of
data
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(687): ajp_read_header:
ajp_ilink_received 03
[Mon Jul 23 10:03:15 2012] [debug] ajp_header.c(697): ajp_parse_type: got 03
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_ajp.c(460): Ignoring flush message
received before headers
[Mon Jul 23 10:03:15 2012] [error] (70014)End of file found:
ajp_ilink_receive() can't receive header
[Mon Jul 23 10:03:15 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_ajp.c(549): (120006)APR does not
understand this error code: ajp_read_header failed
[Mon Jul 23 10:03:15 2012] [debug] mod_proxy_ajp.c(564): proxy: Processing of
request failed backend: 1, output: 0
[Mon Jul 23 10:03:15 2012] [error] (120006)APR does not understand this error
code: proxy: dialog to 127.0.0.1:7071 (127.0.0.1) failed
[Mon Jul 23 10:03:15 2012] [debug] proxy_util.c(2029): proxy: AJP: has released
connection for (127.0.0.1)
[Mon Jul 23 10:03:15 2012] [debug] mod_deflate.c(615): [***cleansed***] Zlib:
Compressed 323 to 218 : URL /myapp/
---------------------------------
Apache HTTP Server (apachectl -V)
---------------------------------
Server version: Apache/2.2.22 (Unix)
Server built: Jul 17 2012 17:20:28
Server's Module Magic Number: 20051115:30
Server loaded: APR 1.4.5, APR-Util 1.4.1
Compiled using: APR 1.4.5, APR-Util 1.4.1
Architecture: 64-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/apps/apache2"
-D SUEXEC_BIN="/apps/apache2/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
-------------------------------
httpd.conf (relevant settings):
-------------------------------
<IfModule worker.c>
StartServers 2
MaxClients 400
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 10000
</IfModule>
ProxyRequests Off
ProxyPreserveHost on
ProxyReceiveBufferSize 0
ProxyErrorOverride Off
RewriteEngine on
ProxyPass /myapp ajp://127.0.0.1:7071/myapp
ProxyPassReverse /myapp ajp://127.0.0.1:7071/myapp
-----------------------------------------------
tcServer 2.0.4.RELEASE (Tomcat 6.0.29 binaries)
-----------------------------------------------
server.xml:
<Executor name="tomcatThreadPool" namePrefix="tomcat-ajp--" maxThreads="300"
minSpareThreads="50"/>
<Connector executor="tomcatThreadPool"
address="127.0.0.1"
port="7071"
protocol="org.apache.coyote.ajp.AjpAprProtocol"
connectionTimeout="60000"
redirectPort="8443"
backlog="100"
tcpNoDelay="true"
enableLookups="false"
URIEncoding="UTF-8" />
