On 07.01.2012 20:22, Sander Temme wrote:
Folks,
I've put in some updates to the mod_heart* modules, taken from the README
supplied with the modules and some perusal of the source code. I have not run
these as I'm too lazy to set up the servers. Review would be appreciated,
especially by the original authors (Cc), before backport to 2.4.x.
I added some comments inline, just based on reading your docs.
Also added a doc file for mod_watchdog. Question for the docs folks: since this one is
only of interest for consumption by other modules, how much documentation should we
supply in the user-facing pages? I said "this module defines programmatic
hooks"... do we need to define what the hook handler prototypes are? That's more
like developer documentation. Thoughts?
I don't know about the history of similar decisions, but IMHO what you
wrote is good enough for 2.4.0.
Modified: httpd/httpd/trunk/docs/manual/mod/mod_heartbeat.xml
URL:
http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/mod/mod_heartbeat.xml?rev=1228700&r1=1228699&r2=1228700&view=diff
==============================================================================
--- httpd/httpd/trunk/docs/manual/mod/mod_heartbeat.xml (original)
+++ httpd/httpd/trunk/docs/manual/mod/mod_heartbeat.xml Sat Jan 7 19:10:28 2012
@@ -35,21 +35,66 @@ that advertises the servers current conn
will be running on a proxy server with<module>mod_lbmethod_heartbeat</module>
loaded, which allows
<directive module="mod_proxy">ProxyPass</directive> to use the
"heartbeat"<em>lbmethod</em> inside
of<directive module="mod_proxy">ProxyPass</directive>.</p>
+<p>
+<module>mod_heartbeat</module> itself is loaded on the origin server(s) that
serve requests
+ through the proxy server(s).
+</p>
+
+<note type="warning">
+ To use<module>mod_heartbeat</module>,
+<module>mod_status</module> and<module>mod_watchdog</module>
+ must be either a static modules or, if a dynamic module, it must
+ be loaded before<module>mod_heartbeat</module>.
+</note>
+
</summary>
+<section id="consuming">
+<title>Consuming mod_heartbeat Output</title>
+<p>
+ Every 1 second, this module generates a single multicast UDP
+ packet, containing the number of busy and idle workers. The
+ packet is a simple ASCII format, similiar to GET query parameters
+ in HTTP.
+</p>
+
+<example><title>An Example Packet</title>
+v=1&ready=75&busy=0
+</example>
+
+<p>
+ Consumers should handle new variables besides busy and ready,
+ separated by '&', being added in the future.
+</p>
+
+</section>
+
+<section id="miscellaneous">
+<title>Miscellaneous</title>
+<p>The interval of 1 seconds is controlled by the HEARTBEAT_INTERVAL
+compile time define. This is not currently tunable at run time. To make
+this module send the status packet more often, you must add to the
+CFLAGS used to compile the module to include:</p>
+<example>-DHEARTBEAT_INTERVAL=3</example>
+<p>Would cause the broadcasts to be sent every 3 seconds.</p>
+</section>
"more often" than 1 second does not match the example "3 seconds" ;)
I guess more often is not possible (if resolution is seconds), so it
would be "less often"?
More important: I think this is totally wrong. The define exists, but I
couldn't find any place in the code, which is influenced by it. I
*guess* it uses the global watchdog interval configured for mod_watchdog.
<directivesynopsis>
<name>HeartbeatAddress</name>
-<description>Address to send heartbeat requests</description>
+<description>Multicast address for heartbeat packets</description>
<syntax>HeartbeatAddress<var>addr:port</var></syntax>
<default>disabled</default>
<contextlist><context>server config</context></contextlist>
<usage>
-<p>The<directive>HeartbeatAddress</directive> directive specifies the
- address<module>mod_heartbeat</module> will send status information to.
This
- address will usually corrspond to a configured<directive
- module="mod_heartmonitor">HeartbeatListen</directive> on a frontend
- proxy system.</p>
+<p>The<directive>HeartbeatAddress</directive> directive specifies the
+multicast address to which<module>mod_heartbeat</module> will send
+status information. This address will usually correspond to a configured
+<directive module="mod_heartmonitor">HeartbeatListen</directive> on a
+frontend proxy system.</p>
+<example>
+HeartbeatAddress 239.0.0.1:27999
+</example>
</usage>
</directivesynopsis>
Should we note, that Multicast needs to be enabled on both notes and
routing needs to be set up, so that the multicast packets from the
origin server can reach the proxy? On the one hand it is obvious, on the
other hand not unlikely to be the typical first problem people run into.
Modified: httpd/httpd/trunk/docs/manual/mod/mod_heartmonitor.xml
URL:
http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/mod/mod_heartmonitor.xml?rev=1228700&r1=1228699&r2=1228700&view=diff
==============================================================================
--- httpd/httpd/trunk/docs/manual/mod/mod_heartmonitor.xml (original)
+++ httpd/httpd/trunk/docs/manual/mod/mod_heartmonitor.xml Sat Jan 7 19:10:28
2012
@@ -20,19 +20,19 @@
limitations under the License.
-->
-<modulesynopsis metafile="mod_heartbeat.xml.meta">
+<modulesynopsis metafile="mod_heartmonitor.xml.meta">
<name>mod_heartmonitor</name>
<description>centralized monitor for mod_heartbeat origin servers</description>
<status>Experimental</status>
-<sourcefile>mod_heartmonitor</sourcefile>
+<sourcefile>mod_heartmonitor.c</sourcefile>
<identifier>heartmonitor_module</identifier>
<compatibility>Available in Apache 2.3 and later</compatibility>
<summary>
<p>
<module>mod_heartmonitor</module> listens for server status messages generated
-by<module>mod_heartbeat</module> enabled servers and makes their status
+by<module>mod_heartbeat</module> enabled origin servers and makes their status
available to<module>mod_lbmethod_heartbeat</module>. This allows
<directive module="mod_proxy">ProxyPass</directive> to use the "heartbeat"
<em>lbmethod</em> inside of<directive module="mod_proxy">ProxyPass</directive>.
@@ -41,22 +41,34 @@ available to<module>mod_lbmethod_heartb
<p>This module uses the services of<module>mod_slotmem_shm</module> when
available instead of flat-file storage. No configuration is required to
use<module>mod_slotmem_shm</module>.</p>
+
+<note type="warning">
+ To use<module>mod_heartmonitor</module>,
+<module>mod_status</module> and<module>mod_watchdog</module>
+ must be either a static modules or, if a dynamic module, it must
+ be loaded before<module>mod_heartmonitor</module>.
+</note>
+
??
</summary>
<directivesynopsis>
<name>HeartbeatListen</name>
-<description>address to listen for incoming heartbeat requests</description>
+<description>multicast address to listen for incoming heartbeat
requests</description>
<syntax>HeartbeatListen<var>addr:port</var></syntax>
<default>disabled</default>
<contextlist><context>server config</context></contextlist>
<usage>
<p>The<directive>HeartbeatListen</directive> directive specifies the
- address the server will listen on for status information from
+ multicast address on which the server will listen for status information
from
<module>mod_heartbeat</module>-enabled servers. This
address will usually corrspond to a configured<directive
module="mod_heartbeat">HeartbeatAddress</directive> on an origin server.
</p>
+
+<example>
+ HeartbeatListen 239.0.0.1:27999
+</example>
Same remark concerning the use of multicast as above.
<p> This module is inactive until this directive is used.</p>
</usage>
@@ -88,7 +100,7 @@ heartbeat requests to this server</descr
<p>The<directive>HeartbeatMaxServers</directive> directive specifies the
maximum number of servers that will be sending requests to this monitor
server. It is used to control the size of the shared memory allocated
- to store the heartbeat info.</p>
+ to store the heartbeat info when<module>mod_slotmem_shm</module> is in
use.</p>
Can this be changed using graceful restart? Only for flat file or also
for shm?
Typo: "corrspond" -> "correspond"
Additional info: it seems the data used for the load balancer is updated
about every 5 seconds (define HM_UPDATE_SEC). More precisely it seems
the heartmonitor is called by watchdog (with interval 0 seconds, but
watchdog sleep at least 100ms), then updates the data from what is had
previously read, then loops for 5 seconds reading new data and then
returns to watchdog. This is done one one thread per child, but a mutex
guarantees, that it only runs in one child at every time. Hope I'm right.
Furthermore the heartmonitor contains some additional code to handle
POST requests directed to it and reading usage data directly from the
POST body (in addition to the multicast method).
I think the current info in mod_lbmethod_heartbeat.xml is confusing. It
contains:
"This modules load balancing algorithm favors servers with more ready
(idle) capacity over time, but does not select the server with the most
ready capacity every time. Servers that have 0 active clients are
penalized, with the assumption that they are not fully initialized."
As far as I can see in the code, it randomly chooses one of the idle
origin server slots based on the last received heartbeat data. Another
equivalent formulation would be: it does a weighted balancing to the
origin servers, where the weights are proportional to the idle (ready)
slots reported with the last heartbeat message.
I couldn't find code that penalizes servers with 0 active clients.
It seems the heartbeat lb opens the data file for each request. So if
I'm right, using slotmem is highly recommended for performance reasons.
</usage>
</directivesynopsis>
</modulesynopsis>
Added: httpd/httpd/trunk/docs/manual/mod/mod_watchdog.xml
URL:
http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/mod/mod_watchdog.xml?rev=1228700&view=auto
==============================================================================
--- httpd/httpd/trunk/docs/manual/mod/mod_watchdog.xml (added)
+++ httpd/httpd/trunk/docs/manual/mod/mod_watchdog.xml Sat Jan 7 19:10:28 2012
@@ -0,0 +1,60 @@
+<?xml version="1.0"?>
+<!DOCTYPE modulesynopsis SYSTEM "../style/modulesynopsis.dtd">
+<?xml-stylesheet type="text/xsl" href="../style/manual.en.xsl"?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<modulesynopsis metafile="mod_watchdog.xml.meta">
+<name>mod_watchdog</name>
+<description>provides infrastructure for other modules to periodically run
+ tasks</description>
+<status>Base</status>
+<sourcefile>mod_watchdog.c</sourcefile>
+<identifier>watchdog_module</identifier>
+<compatibility>Available in Apache 2.3 and later</compatibility>
+
+<summary>
+<p><module>mod_watchdog</module> defines programmatic hooks for other modules
to
+periodically run tasks. These modules can register handlers for
+<module>mod_watchdog</module> hooks. Currently, the following modules in the
+Apache distribution use this functionality:</p>
+<ul>
+<li><module>mod_heartbeat</module></li>
+<li><module>mod_heartmonitor</module></li>
+</ul>
+<note type="warning">
+To allow a module to use<module>mod_watchdog</module> functionality,
+<module>mod_watchdog</module> itself must be statically linked to the server
+core or, if a dynamic module, be loaded before the calling module.
+</note>
+</summary>
+
+<directivesynopsis>
+<name>WatchdogInterval</name>
+<description>Watchdog interval in seconds</description>
+<syntax>WatchdogInterval<var>number-of-seconds</var></syntax>
+<default>WatchdogInterval 1</default>
+<contextlist><context>server config</context></contextlist>
+
+<usage>
+Sets the interval at which the watchdog_step hook runs. Default is to run
every
+second.
+</usage>
+</directivesynopsis>
+</modulesynopsis>
+
We could mention, that there are two ways (and APIs) a module can use
mod_watchdog. Either synchronously called every watchdog interval
seconds, or running on a separate thread called every interval seconds,
where interval is given during the setup.
Furthermore modules can tell mod_watchdog, whether the tasks should be
run in each child independently, or coordinated by a mutex.
Regards,
Rainer
