http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/list_of_mbeans_full.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/list_of_mbeans_full.html.md.erb b/geode-docs/managing/management/list_of_mbeans_full.html.md.erb new file mode 100644 index 0000000..74fb2bd --- /dev/null +++ b/geode-docs/managing/management/list_of_mbeans_full.html.md.erb @@ -0,0 +1,210 @@ +--- +title: JMX Manager MBeans +--- +<a id="topic_14E3721DD0CF47D7AD8C742DFBE9FB9C"></a> + + +This section describes the MBeans that are available on the JMX Manager node. + +The JMX Manager node includes all local beans listed under [Managed Node MBeans](list_of_mbeans_full.html#topic_48194A5BDF3F40F68E95A114DD702413) and the following beans that are available only on the JMX Manager node: + +- [ManagerMXBean](list_of_mbeans_full.html#topic_14E3721DD0CF47D7AD8C742DFBE9FB9C__section_7B878B450B994514BDFE96571F0D3827) +- [DistributedSystemMXBean](list_of_mbeans_full.html#topic_14E3721DD0CF47D7AD8C742DFBE9FB9C__section_4D7A4C82DD974BB5A5E52B34A6D888B4) +- [DistributedRegionMXBean](list_of_mbeans_full.html#topic_14E3721DD0CF47D7AD8C742DFBE9FB9C__section_48384B091AB846E591F22EEA2770DD36) +- [DistributedLockServiceMXBean](list_of_mbeans_full.html#topic_14E3721DD0CF47D7AD8C742DFBE9FB9C__section_9E004D8AA3D24647A5C19CAA1879F0A4) + +## <a id="topic_14E3721DD0CF47D7AD8C742DFBE9FB9C__section_7B878B450B994514BDFE96571F0D3827" class="no-quick-link"></a>ManagerMXBean + +Represents the Geode Management layer for the hosting member. Controls the scope of management. This MBean provides `start` and `stop` methods to turn a managed node into a JMX Manager node or to stop a node from being a JMX Manager. For potential managers (`jmx-manager=true` and `jmx-manager-start=false`), this MBean is created when a Locator requests it. + +**Note:** +You must configure the node to allow it to become a JMX Manager. See [Configuring a JMX Manager](jmx_manager_operations.html#topic_263072624B8D4CDBAD18B82E07AA44B6) for configuration information. + +**MBean Details** + +| | | +|--------------------|----------------------------------------------------------------------------| +| Scope | ALL | +| Proxied | No | +| Object Name | GemFire:type=Member, service=Manager,member=<name-or-dist-member-id> | +| Instances Per Node | 1 | + +See the `org.apache.geode.management.ManagerMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_14E3721DD0CF47D7AD8C742DFBE9FB9C__section_4D7A4C82DD974BB5A5E52B34A6D888B4" class="no-quick-link"></a>DistributedSystemMXBean + +System-wide aggregate MBean that provides a high-level view of the entire distributed system including all members (cache servers, peers, locators) and their caches. At any given point of time, it can provide a snapshot of the complete distributed system and its operations. + +The DistributedSystemMXBean provides APIs for performing distributed system-wide operations such as backing up all members, shutting down all members or showing various distributed system metrics. + +You can attach a standard JMX NotificationListener to this MBean to listen for notifications throughout the distributed system. See [Geode JMX MBean Notifications](mbean_notifications.html) for more information. + +This MBean also provides some MBean model navigation APIS. These APIs should be used to navigate through all the MBeans exposed by a Geode System. + +**MBean Details** + +| | | +|--------------------|-----------------------------------------| +| Scope | Aggregate | +| Proxied | No | +| Object Name | GemFire:type=Distributed,service=System | +| Instances Per Node | 1 | + +See the `org.apache.geode.management.DistributedSystemMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_14E3721DD0CF47D7AD8C742DFBE9FB9C__section_48384B091AB846E591F22EEA2770DD36" class="no-quick-link"></a>DistributedRegionMXBean + +System-wide aggregate MBean of a named region. It provides a high-level view of a region for all members hosting and/or using that region. For example, you can obtain a list of all members that are hosting the region. Some methods are only available for partitioned regions. + +**MBean Details** + +| | | +|--------------------|-----------------------------------------------------------------| +| Scope | Aggregate | +| Proxied | No | +| Object Name | GemFire:type=Distributed,service=Region,name=<regionName> | +| Instances Per Node | 0..N | + +See the `org.apache.geode.management.DistributedRegionMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_14E3721DD0CF47D7AD8C742DFBE9FB9C__section_9E004D8AA3D24647A5C19CAA1879F0A4" class="no-quick-link"></a>DistributedLockServiceMXBean + +Represents a named instance of DistributedLockService . Any number of DistributedLockService can be created in a member. + +A named instance of DistributedLockService defines a space for locking arbitrary names across the distributed system defined by a specified distribution manager. Any number of DistributedLockService instances can be created with different service names. For all processes in the distributed system that have created an instance of DistributedLockService with the same name, no more than one thread is permitted to own the lock on a given name in that instance at any point in time. Additionally, a thread can lock the entire service, preventing any other threads in the system from locking the service or any names in the service. + +**MBean Details** + +| | | +|--------------------|-------------------------------------------------------------------| +| Scope | Aggregate | +| Proxied | No | +| Object Name | GemFire:type=Distributed,service=LockService,name=<dlsName> | +| Instances Per Node | 0..N | + +See the `org.apache.geode.management.DistributedLockServiceMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_48194A5BDF3F40F68E95A114DD702413" class="no-quick-link"></a>Managed Node MBeans + +This section describes the MBeans that are available on all managed nodes. + +MBeans that are available on all managed nodes include: + +- [MemberMXBean](list_of_mbeans_full.html#topic_48194A5BDF3F40F68E95A114DD702413__section_796A989549304BF7A536A33A913322A4) +- [CacheServerMXBean](list_of_mbeans_full.html#topic_48194A5BDF3F40F68E95A114DD702413__section_7287A7560650426E9B8249E2D87CE55F) +- [RegionMXBean](list_of_mbeans_full.html#topic_48194A5BDF3F40F68E95A114DD702413__section_577A666924E54352AF69294DC8DEFEBF) +- [LockServiceMXBean](list_of_mbeans_full.html#topic_48194A5BDF3F40F68E95A114DD702413__section_2F9F00081BB14CE0ADA251F5B6289BF2) +- [DiskStoreMXBean](list_of_mbeans_full.html#topic_48194A5BDF3F40F68E95A114DD702413__section_1F475F68E73B4EAE875BA40825E736C9) +- [AsyncEventQueueMXBean](list_of_mbeans_full.html#topic_48194A5BDF3F40F68E95A114DD702413__section_6A77030A15704BFEAEBBD7DB88266BF6) +- [LocatorMXBean](list_of_mbeans_full.html#topic_48194A5BDF3F40F68E95A114DD702413__section_BB83107990D346F39271ACCC14CB84A0) + +JMX Manager nodes will have managed node MBeans for themselves since they are also manageable entities in the distributed system. + +## <a id="topic_48194A5BDF3F40F68E95A114DD702413__section_796A989549304BF7A536A33A913322A4" class="no-quick-link"></a>MemberMXBean + +Member's local view of its connection and cache. It is the primary gateway to manage a particular member. It exposes member level attributes and statistics. Some operations like `createCacheServer()` and `createManager()` will help to create some Geode resources. Any JMX client can connect to the MBean server and start managing a Geode Member by using this MBean. + +See [MemberMXBean Notifications](list_of_mbean_notifications.html#reference_czt_hq2_vj) for a list of notifications emitted by this MBean. + +**MBean Details** + +| | | +|--------------------|-----------------------------------------------------------| +| Scope | Local | +| Proxied | Yes | +| Object Name | GemFire:type=Member,member=<name-or-dist-member-id> | +| Instances Per Node | 1 | + +See the `org.apache.geode.management.MemberMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_48194A5BDF3F40F68E95A114DD702413__section_7287A7560650426E9B8249E2D87CE55F" class="no-quick-link"></a>CacheServerMXBean + +Represents the Geode CacheServer. Provides data and notifications about server, subscriptions, durable queues and indices. + +See [CacheServerMXBean Notifications](list_of_mbean_notifications.html#cacheservermxbean_notifications) for a list of notifications emitted by this MBean. + +**MBean Details** + +| | | +|--------------------|-------------------------------------------------------------------------------| +| Scope | Local | +| Proxied | Yes | +| Object Name | GemFire:type=Member,service=CacheServer,member=<name-or-dist-member-id> | +| Instances Per Node | 1 | + +See the `org.apache.geode.management.CacheServerMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_48194A5BDF3F40F68E95A114DD702413__section_577A666924E54352AF69294DC8DEFEBF" class="no-quick-link"></a>RegionMXBean + +Member's local view of region. + +**MBean Details** + +| | | +|--------------------|--------------------------------------------------------------------------------------------------| +| Scope | Local | +| Proxied | Yes | +| Object Name | GemFire:type=Member,service=Region,name=<regionName>,member=<name-or-dist-member-id> | +| Instances Per Node | 0..N | + +See the `org.apache.geode.management.RegionMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_48194A5BDF3F40F68E95A114DD702413__section_2F9F00081BB14CE0ADA251F5B6289BF2" class="no-quick-link"></a>LockServiceMXBean + +Represents a named instance of a LockService . Any number of LockServices can be created in a member. + +**MBean Details** + +| | | +|--------------------|----------------------------------------------------------------------------------------------------| +| Scope | Local | +| Proxied | Yes | +| Object Name | GemFire:type=Member,service=LockService,name=<dlsName>,member=<name-or-dist-member-id> | +| Instances Per Node | 0..N | + +See the `org.apache.geode.management.LockServiceMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_48194A5BDF3F40F68E95A114DD702413__section_1F475F68E73B4EAE875BA40825E736C9" class="no-quick-link"></a>DiskStoreMXBean + +Represents a DiskStore object which provides disk storage for one or more regions + +**MBean Details** + +| | | +|--------------------|-----------------------------------------------------------------------------------------------| +| Scope | Local | +| Proxied | Yes | +| Object Name | GemFire:type=Member,service=DiskStore,name=<name>,member=<name-or-dist-member-id> | +| Instances Per Node | 0..N | + +See the `org.apache.geode.management.DiskStoreMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_48194A5BDF3F40F68E95A114DD702413__section_6A77030A15704BFEAEBBD7DB88266BF6" class="no-quick-link"></a>AsyncEventQueueMXBean + +An AsyncEventQueueMXBean provides access to an AsyncEventQueue, which represent the channel over which events are delivered to the AsyncEventListener. + +**MBean Details** + +| | | +|--------------------|------------------------------------------------------------------------------------------------------------| +| Scope | Local | +| Proxied | Yes | +| Object Name | GemFire:type=Member,service=AsyncEventQueue=<AsyncEventQueue> ,member=<name-or-dist-member-id> | +| Instances Per Node | 0..N | + +See the `org.apache.geode.management.AsyncEventQueueMXBean` JavaDocs for information on available MBean methods and attributes. + +## <a id="topic_48194A5BDF3F40F68E95A114DD702413__section_BB83107990D346F39271ACCC14CB84A0" class="no-quick-link"></a>LocatorMXBean + +A LocatorMXBean represents a locator . + +**MBean Details** + +| | | +|--------------------|---------------------------------------------------------------------------------------------| +| Scope | Local | +| Proxied | Yes | +| Object Name | GemFire:type=Member,service=Locator,port=<port>,member=<name-or-dist-member-id> | +| Instances Per Node | 0..1 | + +See the `org.apache.geode.management.LocatorMXBean` JavaDocs for information on available MBean methods and attributes.
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/management_and_monitoring.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/management_and_monitoring.html.md.erb b/geode-docs/managing/management/management_and_monitoring.html.md.erb new file mode 100644 index 0000000..f0b57e6 --- /dev/null +++ b/geode-docs/managing/management/management_and_monitoring.html.md.erb @@ -0,0 +1,35 @@ +--- +title: Apache Geode Management and Monitoring +--- + +Apache Geode provides APIs and tools for managing your distributed system and monitoring the health of your distributed system members. + +- **[Management and Monitoring Features](../../managing/management/management_and_monitoring_features.html)** + + Apache Geode uses a federated Open MBean strategy to manage and monitor all members of the distributed system. This strategy gives you a consolidated, single-agent view of the distributed system. + +- **[Overview of Geode Management and Monitoring Tools](../../managing/management/mm_overview.html)** + + Geode provides a variety of management tools you can use to manage a Geode distributed system. + +- **[Architecture and Components](../../managing/management/management_system_overview.html)** + + Geode's management and monitoring system consists of one JMX Manager node (there should only be one) and one or more managed nodes within a distributed system. All members in the distributed system are manageable through MBeans and Geode Management Service APIs. + +- **[JMX Manager Operations](../../managing/management/jmx_manager_node.html#topic_36C918B4202D45F3AC225FFD23B11D7C)** + + Any member can host an embedded JMX Manager, which provides a federated view of all MBeans for the distributed system. The member can be configured to be a manager at startup or anytime during its life by invoking the appropriate API calls on the ManagementService. + +- **[Federated MBean Architecture](../../managing/management/mbean_architecture.html)** + + Geode uses MBeans to manage and monitor different parts of Geode. Geode's federated MBean architecture is scalable and allows you to have a single-agent view of a Geode distributed system. + +- **[Configuring RMI Registry Ports and RMI Connectors](../../managing/management/configuring_rmi_connector.html)** + + Geode programmatically emulates out-of-the-box JMX provided by Java and creates a JMXServiceURL with RMI Registry and RMI Connector ports on all manageable members. + +- **[Executing gfsh Commands through the Management API](../../managing/management/gfsh_and_management_api.html)** + + You can also use management APIs to execute gfsh commands programmatically. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/management_and_monitoring_features.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/management_and_monitoring_features.html.md.erb b/geode-docs/managing/management/management_and_monitoring_features.html.md.erb new file mode 100644 index 0000000..bbd0c29 --- /dev/null +++ b/geode-docs/managing/management/management_and_monitoring_features.html.md.erb @@ -0,0 +1,24 @@ +--- +title: Management and Monitoring Features +--- + +Apache Geode uses a federated Open MBean strategy to manage and monitor all members of the distributed system. This strategy gives you a consolidated, single-agent view of the distributed system. + +<a id="concept_F7B9EE348DA744D3BBDFD68E7F48A604__section_37CECE9B26644505A79784EA0CD1FDAE"></a> +Application and manager development is much easier because you do not have to find the right MBeanServer to make a request on an MBean. Instead, you interact with a single MBeanServer that aggregates MBeans from all other local and remote MBeanServers. + +Some other key advantages and features of Geode administration architecture: + +- Geode monitoring is tightly integrated into Geode's processes instead of running in a separately installed and configured monitoring agent. You can use the same framework to actually manage Geode and perform administrative operations, not just monitor it. +- All Geode MBeans are *MXBeans*. They represent useful and relevant information on the state of the distributed system and all its members. Because MXBeans use the Open MBean model with a predefined set of types, clients and remote management programs no longer require access to model-specific classes representing your MBean types. Using MXBeans adds flexibility to your selection of clients and makes the Geode management and monitoring much easier to use. +- Each member in the distributed system is manageable through MXBeans, and each member hosts its own MXBeans in a Platform MBeanServer. +- Any Geode member can be configured to provide a federated view of all the MXBeans for all members in a Geode cluster. +- Geode has also modified its use of JMX to be industry-standard and friendly to generic JMX clients. You can now easily monitor or manage the distributed system by using any third-party tool that is compliant with JMX. For example, JConsole. + +## <a id="concept_F7B9EE348DA744D3BBDFD68E7F48A604__section_A3166A9657044E088DA0FE2C2B8325BE" class="no-quick-link"></a>References + +For more information on MXBeans and Open MBeans, see: + +- [http://docs.oracle.com/javase/8/docs/api/javax/management/MXBean.html](http://docs.oracle.com/javase/8/docs/api/javax/management/MXBean.html) +- [http://docs.oracle.com/javase/8/docs/api/javax/management/openmbean/package-summary.html](http://docs.oracle.com/javase/8/docs/api/javax/management/openmbean/package-summary.html) + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/management_system_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/management_system_overview.html.md.erb b/geode-docs/managing/management/management_system_overview.html.md.erb new file mode 100644 index 0000000..33809b0 --- /dev/null +++ b/geode-docs/managing/management/management_system_overview.html.md.erb @@ -0,0 +1,95 @@ +--- +title: Architecture and Components +--- + +Geode's management and monitoring system consists of one JMX Manager node (there should only be one) and one or more managed nodes within a distributed system. All members in the distributed system are manageable through MBeans and Geode Management Service APIs. + +## <a id="concept_1BAE2CE1146B4347ABD61F50B9F9781F__section_ABE7007BE3C244FBA0418C4B5BE7E1F2" class="no-quick-link"></a>Architecture + +The following diagram depicts the architecture of the management and monitoring system components. + +<img src="../../images_svg/JMX_Architecture.svg" id="concept_1BAE2CE1146B4347ABD61F50B9F9781F__image_1E9E8575E13D4087BC47B6A288097B7A" class="image" /> + +In this architecture every Geode member is manageable. All Geode MBeans for the local Geode processes are automatically registered in the Platform MBeanServer (the default MBeanServer of each JVM that hosts platform MXBeans.) + +## <a id="concept_1BAE2CE1146B4347ABD61F50B9F9781F__section_1CF2B237C16F4095A609E62F0C7146C1" class="no-quick-link"></a>Managed Node + +Each member of a distributed system is a managed node. Any node that is not currently also acting as a JMX Manager node is referred to simply as a managed node. A managed node has the following resources so that it can answer JMX queries both locally and remotely: + +- Local MXBeans that represent the locally monitored components on the node. See [List of Geode JMX MBeans](list_of_mbeans.html#topic_4BCF867697C3456D96066BAD7F39FC8B) for a list of possible MXBeans existing for the managed node. +- Built-in platform MBeans. + +## <a id="concept_1BAE2CE1146B4347ABD61F50B9F9781F__section_8604838507194C8B86F1420FBA46894C" class="no-quick-link"></a>JMX Manager Node + +A JMX Manager node is a member that can manage other Geode members --that is, other managed nodes -- as well as itself. A JMX Manager node can manage all other members in the distributed system. + +To convert a managed node to a JMX Manager node, you configure the Geode property `jmx-manager=true`, in the `gemfire.properties` file, and start the member as a JMX Manager node. + +You start the member as a JMX Manager node when you provide` --J=-Dgemfire.jmx-manager=true` as an argument to either the` start server` or `start locator` command. See [Starting a JMX Manager](jmx_manager_operations.html#topic_686158E9AFBD47518BE1B4BEB232C190) for more information. + +The JMX Manager node has the following extra resources allocated so that it can answer JMX queries: + +- RMI connector that allows JMX clients to connect to and access all MXBeans in the distributed system. +- Local MXBeans that represent the locally monitored components on this node, same as any other managed node. +- Aggregate MXBeans: + - DistributedSystemMXBean + - DistributedRegionMXBean + - DistributedLockServiceMXBean +- ManagerMXBean with Scope=ALL, which allows various distributed system-wide operations. +- Proxy to MXBeans on managed nodes. +- Built-in platform MXBeans. + +## <a id="concept_1BAE2CE1146B4347ABD61F50B9F9781F__section_32D9F98189B14AA09BAC5E843EC18EDA" class="no-quick-link"></a>JMX Integration + +Management and monitoring tools such as gfsh command-line interface and Pulse use JMX/RMI as the communication layer to connect to Geode nodes. All Geode processes by default allow JMX connections to the Platform MBeanServer from localhost. By default, both managed nodes and JMX manager nodes have RMI connectors enabled to allow JMX client connections. + +JConsole (and other similar JMX clients that support Sun's Attach API) can connect to any local JVM without requiring an RMI connector by using the Attach API. This allows connections from the same machine. + +JConsole (and other JMX clients) can connect to any JVM if that JVM is configured to start an RMI connector. This allows remote connections from other machines. + +JConsole can connect to any Geode member, but if it connects to a non-JMX-Manager member, JConsole only detects the local MBeans for the node, and not MBeans for the cluster. + +When a Geode locator or server becomes a JMX Manager for the cluster, it enables the RMI connector. JConsole can then connect only to that one JVM to view the MBeans for the entire cluster. It does not need to connect to all the other JVMs. Geode manages the inter-JVM communication required to provide a federated view of all MBeans in the distributed system. + +`gfsh` can only connect to a JMX Manager or to a locator. If connected to a locator, the locator provides the necessary connection information for the existing JMX Manager. If the locator detects a JMX Manager is not already running in the cluster, the locator makes itself a JMX Manager. gfsh cannot connect to other non-Manager or non-locator members. + +For information on how to configure the RMI registry and RMI connector, see [Configuring RMI Registry Ports and RMI Connectors](configuring_rmi_connector.html#concept_BC793A7ACF9A4BD9A29C2DCC6894767D). + +## <a id="concept_1BAE2CE1146B4347ABD61F50B9F9781F__section_A3F9E1594982480DA019CBA3E93CA895" class="no-quick-link"></a>Management APIs + +Geode management APIs represent the Geode cluster to a JMX user. However, they do not provide functionality that is otherwise present in JMX. They only provide a gateway into various services exclusively offered by Geode monitoring and management. + +The entry point to Geode management is through the ManagementService interface. For example, to create an instance of the Management Service: + +``` pre +ManagementService service = ManagementService.getManagementService(cache); +``` + +The resulting ManagementService instance is specific to the provided cache and its distributed system. The implementation of getManagementService is a singleton for now but may eventually support multiple cache instances. + +You can use the Geode management APIs to accomplish the following tasks: + +- Monitor the health status of clients. +- Obtain the status and results of individual disk backups. +- View metrics related to disk usage and performance for a particular member. +- Browse Geode properties set for a particular member. +- View JVM metrics such as memory, heap, and thread usage. +- View network metrics, such as bytes received and sent. +- View partition region attributes such as total number of buckets, redundant copy, and maximum memory information. +- View persistent member information such as disk store ID. +- Browse region attributes. + +See the JavaDocs for the `org.apache.geode.management` package for more details. + +You can also execute gfsh commands using the ManagementService API. See [Executing gfsh Commands through the Management API](gfsh_and_management_api.html#concept_451F0978285245E69C3E8DE795BD8635) and the JavaDocs for the `org.apache.geode.management.cli` package. + +## <a id="concept_1BAE2CE1146B4347ABD61F50B9F9781F__section_E69A93A6309E4747B52850D81FE1674E" class="no-quick-link"></a>Geode Management and Monitoring Tools + +This section lists the currently available tools for managing and monitoring Geode: + +- **gfsh**. Apache Geode command-line interface that provides a simple & powerful command shell that supports the administration, debugging and deployment of Geode applications. It features context sensitive help, scripting and the ability to invoke any commands from within the application using a simple API. See [gfsh](../../tools_modules/gfsh/chapter_overview.html). +- **Geode Pulse**. Easy-to-use, browser-based dashboard for monitoring Geode deployments. Geode Pulse provides an integrated view of all Geode members within a distributed system. See [Geode Pulse](../../tools_modules/pulse/chapter_overview.html). +- **Pulse Data Browser**. This Geode Pulse utility provides a graphical interface for performing OQL ad-hoc queries in a Geode distributed system. See [Data Browser](../../tools_modules/pulse/quickstart.html#topic_F0ECE9E8179541CCA3D6C5F4FBA84404__sec_pulsedatabrowser). +- **Other Java Monitoring Tools such as JConsole and jvisualvm.** JConsole is a JMX-based management and monitoring tool provided in the Java 2 Platform that provides information on the performance and consumption of resources by Java applications. See [http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html](http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html). **Java VisualVM (jvisualvm)** is a profiling tool for analyzing your Java Virtual Machine. Java VisualVM is useful to Java application developers to troubleshoot applications and to monitor and improve the applications' performance. Java VisualVM can allow developers to generate and analyse heap dumps, track down memory leaks, perform and monitor garbage collection, and perform lightweight memory and CPU profiling. For more details on using jvisualvm, see [http://docs.oracle.com/javase/6/docs/technotes/tools/share/jvisualvm.html](http://docs.oracle.com/javase/6/docs/technot es/tools/share/jvisualvm.html). + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/mbean_architecture.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/mbean_architecture.html.md.erb b/geode-docs/managing/management/mbean_architecture.html.md.erb new file mode 100644 index 0000000..d966cb6 --- /dev/null +++ b/geode-docs/managing/management/mbean_architecture.html.md.erb @@ -0,0 +1,59 @@ +--- +title: Federated MBean Architecture +--- + +Geode uses MBeans to manage and monitor different parts of Geode. Geode's federated MBean architecture is scalable and allows you to have a single-agent view of a Geode distributed system. + +## <a id="concept_40A475F186E249C597681069C835CF65__section_19948055E4184110910B11CD979A923A" class="no-quick-link"></a>Federation of Geode MBeans and MBeanServers + +Federation of the MBeanServers means that one member, the JMX Manager Node, can provide a proxied view of all the MBeans that the MBeanServer hosts. Federation also means that operations and notifications are spread across the distributed system. + +Geode federation takes care of the following functionality: + +- MBean proxy creation +- MBean state propagation +- Notifications propagation +- Operation invocation + +## <a id="concept_40A475F186E249C597681069C835CF65__section_AD13594ADA814194897488CF96BCC479" class="no-quick-link"></a>MBean Proxy Naming Conventions + +Each Geode MBean follows a particular naming convention for easier grouping. For example: + +``` pre +GemFire:type=Member,service=LockService,name=<dlsName>,memberName=<memberName> +``` + +At the JMX Manager node, this MBean will be registered with GemFire/<memberId> as domain. + +The following are some sample MBean names: + +MemberMBean: + +``` pre +GemFire:type=Member,member=<Node1> +``` + +## <a id="concept_40A475F186E249C597681069C835CF65__section_8F9D375A185E476FB50E7D6E30BE2FC7" class="no-quick-link"></a>Use of MXBeans + +In its Management API, Geode provides MXBeans to ensure that any MBeans that are created are usable by any client, including remote clients, without requiring the client to access specific classes in order to access contents of the MBean. + +## <a id="concept_40A475F186E249C597681069C835CF65__section_DCC1B2AB80B04E8CBED041C1F3BDAB5F" class="no-quick-link"></a>MBean Proxy Creation + +Geode proxies are inherently local MBeans. Every Geode JMX manager member hosts proxies pointing to the local MBeans of every managed node. Proxy MBeans will also emit any notification emitted by local MBeans in managed nodes when an event occurs in that managed node. + +**Note:** +Aggregate MBeans on the JMX Manager node are not proxied. + +- **[List of Geode JMX MBeans](../../managing/management/list_of_mbeans.html)** + + This topic provides descriptions for the various management and monitoring MBeans that are available in Geode. + +- **[Browsing Geode MBeans through JConsole](../../managing/management/mbeans_jconsole.html)** + + You can browse all the Geode MBeans in your distributed system by using JConsole. + +- **[Geode JMX MBean Notifications](../../managing/management/mbean_notifications.html)** + + Apache Geode MBeans emit notifications when specific events occur or if an alert is raised in the Geode system. Using standard JMX APIs, users can add notification handlers to listen for these events. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/mbean_notifications.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/mbean_notifications.html.md.erb b/geode-docs/managing/management/mbean_notifications.html.md.erb new file mode 100644 index 0000000..34b40d3 --- /dev/null +++ b/geode-docs/managing/management/mbean_notifications.html.md.erb @@ -0,0 +1,17 @@ +--- +title: Geode JMX MBean Notifications +--- +<a id="topic_czt_hq2_vk"></a> + + +Apache Geode MBeans emit notifications when specific events occur or if an alert is raised in the Geode system. Using standard JMX APIs, users can add notification handlers to listen for these events. + +- **[Notification Federation](notification_federation_and_alerts.html)** + + All notifications emitted from managed nodes are federated to all JMX Managers in the system. + +- **[List of JMX MBean Notifications](list_of_mbean_notifications.html)** + + This topic lists all available JMX notifications emitted by Geode MBeans. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/mbeans_jconsole.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/mbeans_jconsole.html.md.erb b/geode-docs/managing/management/mbeans_jconsole.html.md.erb new file mode 100644 index 0000000..3a9f141 --- /dev/null +++ b/geode-docs/managing/management/mbeans_jconsole.html.md.erb @@ -0,0 +1,36 @@ +--- +title: Browsing Geode MBeans through JConsole +--- + +You can browse all the Geode MBeans in your distributed system by using JConsole. + +To view Geode MBeans through JConsole, perform the following steps: + +1. Start a `gfsh` prompt. +2. Connect to a running distributed system by either connecting to a locator with an embedded JMX Manager or connect directly to a JMX Manager. For example: + + ``` pre + gfsh>connect --locator=locator1[10334] + ``` + + or + + ``` pre + gfsh>connect --jmx-manager=locator1[1099] + ``` + +3. Start JConsole: + + ``` pre + gfsh>start jconsole + ``` + + If successful, the message `Running JDK JConsole` appears. The JConsole application launches and connects directly to the JMX Manager using RMI. + +4. On the JConsole screen, click on the MBeans tab. Expand **GemFire**. Then expand each MBean to browse individual MBean attributes, operations and notifications. + + The following is an example screenshot of the MBean hierarchy in a Geode distributed system: + + <img src="../../images/jconsole_mbeans.png" id="concept_492532E145834248997BD23BCAC7AD45__image_7A45BE69B67A44A7A8AD40343A2B0AEB" class="image" /> + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/mm_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/mm_overview.html.md.erb b/geode-docs/managing/management/mm_overview.html.md.erb new file mode 100644 index 0000000..b21c7d1 --- /dev/null +++ b/geode-docs/managing/management/mm_overview.html.md.erb @@ -0,0 +1,77 @@ +--- +title: Overview of Geode Management and Monitoring Tools +--- + +Geode provides a variety of management tools you can use to manage a Geode distributed system. + +The Geode management and monitoring tools allow you to configure all members and processes of a distributed system, monitor operations in the system, and start and stop the members. Internally, Geode uses Java MBeans, specifically MXBeans, to expose management controls and monitoring features. You can monitor and control Geode by writing Java programs that use these MXBeans, or you can use one of several tools provided with Geode to monitor and manage your distributed system. The primary tool for these tasks is the gfsh command-line tool, as described in this section. + +Geode provides the following tools to manage a Geode installation: + +## gfsh Command-line tool + +The gfsh command line tool provides a set of commands you use to configure, manage, and monitor a Geode distributed system. gfsh is the recommended tool for managing your distributed system. + +Use gfsh to: + +- Start and stop Geode processes, such as locators and cache servers +- Deploy applications +- Create and destroy regions +- Execute functions +- Manage disk stores +- Import and export data +- Monitor Geode processes +- Launch Geode monitoring tools +- Shut down a distributed system +- Script various operations involving Geode members +- Save the configuration for all members of a distributed system + +gfsh runs in its own shell, or you can [execute gfsh commands directly from the OS command line](../../tools_modules/gfsh/os_command_line_execution.html#topic_fpf_y1g_tp). gfsh can interact with remote systems [using the http protocol](../../configuring/cluster_config/gfsh_remote.html). You can also [write scripts that run in a gfsh shell](../../tools_modules/gfsh/command_scripting.html#concept_9B2F7550F16C4717831AD40A56922259) to automate system startup. + +You can use gfsh to create shared cluster configurations for your distributed system. You can define configurations that apply to the entire cluster, or that apply only to groups of similar members that all share a common configuration. Geode locators maintain these configurations as a hidden region and distribute the configuration to all locators in the distributed system. The locator also persists the shared configurations on disk as `cluster.xml` and `cluster.properties` files. You can use those shared cluster configuration files to re-start your system, migrate the system to a new environment, add new members to a distributed system, or to restore existing members after a failure. + +A basic cluster configuration consists of: + +- `cluster.xml` file shared by the cluster +- `cluster.properties` file shared by the cluster +- Deployed jar files containing application Java classes. + +See [Overview of the Cluster Configuration Service](../../configuring/cluster_config/gfsh_persist.html) and [Cluster Configuration Files and Troubleshooting](../../configuring/cluster_config/gfsh_config_troubleshooting.html#concept_ylt_2cb_y4) for additional details on gfsh cluster configuration files. + +Using the gfsh tool, you can easily migrate a Geode-based application from a development environment into a testing or production environment. + +## Executing gfsh commands with the management API + +You can also use Geode's management APIs to execute gfsh commands in a Java class. See [Executing gfsh Commands through the Management API](gfsh_and_management_api.html#concept_451F0978285245E69C3E8DE795BD8635). + +## Member Configuration Management + +When you issue gfsh commands and have the cluster configuration service enabled (on a locator), Geode saves the configurations created within gfsh by building a `cluster.xml` and `cluster.properties` files for the entire cluster, or group of members. + +You can also directly create configurations using `cache.xml` and `gemfire.properties` files and manage the members individually. + +## Java Management Extension (JMX) MBeans + +Geode uses a federated Open MBean strategy to manage and monitor all members of the distributed system. Your Java classes interact with a single MBeanServer that aggregates MBeans from other local and remote members. Using this strategy gives you a consolidated, single-agent view of the distributed system. + +Geode's implementation of JMX is industry-standard and friendly to generic JMX clients. You can monitor or manage the distributed system by using any third-party tool that is compliant with JMX. For example, JConsole. + +See [Apache Geode Management and Monitoring](management_and_monitoring.html) + +## Geode Java API + +The Geode API provides a set of Java classes you can use to manage and monitor a distributed system. See the <span class="keyword apiname">org.apache.geode.management</span> package in the Geode JavaDocs . + +## Geode Pulse + +Geode Pulse is a Web Application that provides a graphical dashboard for monitoring vital, real-time health and performance of Geode clusters, members, and regions. + +Use Pulse to examine total memory, CPU, and disk space used by members, uptime statistics, client connections, and critical notifications. Pulse communicates with a Geode JMX manager to provide a complete view of your Geode deployment. + +See [Geode Pulse](../../tools_modules/pulse/chapter_overview.html). + +## JConsole + +JConsole is a JMX monitoring utility provided with a Java Development Kit (JDK). You use gfsh to connect to Geode, and then launch JConsole with a gfsh command. The JConsole application allows you to browse MBeans, attributes, operations, and notifications. See [Browsing Geode MBeans through JConsole](mbeans_jconsole.html). + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/notification_federation_and_alerts.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/notification_federation_and_alerts.html.md.erb b/geode-docs/managing/management/notification_federation_and_alerts.html.md.erb new file mode 100644 index 0000000..ed56825 --- /dev/null +++ b/geode-docs/managing/management/notification_federation_and_alerts.html.md.erb @@ -0,0 +1,37 @@ +--- +title: Notification Federation +--- + +All notifications emitted from managed nodes are federated to all JMX Managers in the system. + +These notifications are federated and then emitted by the DistributedSystemMXBean. If you attach a `javax.management.NotificationListener` to your DistributedSystemMXBean, the NotificationListener can listen to notifications from all MemberMXBeans and all CacheServerMXBeans. + +## <a id="topic_212EE5A2ABAB4E8E8EF71807C9ECEF1A__section_2909371D90764736B3AC7BE9E4BC1BE4" class="no-quick-link"></a>Attaching Listeners to MXBeans + +When you attach a notification listener to the DistributedSystemMXBean, the DistributedSystemMXBean then acts as the notification hub for the entire distributed system. You do not have to attach a listener to each individual member or cache server MBean in order to listen to all the notifications in the distributed system. + +The following is an example of attaching a NotificationListener to an MBean using the JMX MBeanServer API: + + NotificationListener myListener = ... + ObjectName mbeanName = ... + MBeanServer.addNotificationListener(mbeanName, myListener, null, null); + + +JMX Managers will emit notifications for all distributed system members with two exceptions: + +- If you use cache.xml to define resources such as regions and disks, then notifications for these resources are not federated to the JMX Manager. In those cases, the DistributedSystemMXBean cannot emit those notifications. +- If a JMX Manager is started after a resource has been created, the JMX Manager cannot emit notifications for that resource. + +## <a id="topic_212EE5A2ABAB4E8E8EF71807C9ECEF1A__section_7463D13112D54406953416356835E290" class="no-quick-link"></a>System Alert Notifications + +System alerts are Geode alerts wrapped within a JMX notification. The JMX Manager registers itself as an alert listener with each member of the system, and by default, it receives all messages logged with the SEVERE alert level by any node in the distributed system. Consequently, the DistributedSystemMXBean will then emit notifications for these alerts on behalf of the DistributedSystem. + +By default, the JMX Manager registers itself to send notifications only for SEVERE level alerts. To change the alert level that the JMX Manager will send notifications for, use the `DistributedMXBean.changeAlertLevel` method. Possible alert levels to set are WARNING, ERROR, SEVERE, and NONE. After changing the level, the JMX Manager will only emit that level of log message as notifications. + +Notification objects include **type**, **source** and **message** attributes. System alerts also include the **userData** attribute. For system alerts, the notification object attributes correspond to the following: + +- **type**: system.alert +- **source**: Distributed System ID +- **message**: alert message +- **userData**: name or ID of the member that raised the alert + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/management/programming_example.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/management/programming_example.html.md.erb b/geode-docs/managing/management/programming_example.html.md.erb new file mode 100644 index 0000000..45e4ac5 --- /dev/null +++ b/geode-docs/managing/management/programming_example.html.md.erb @@ -0,0 +1,220 @@ +--- +title: Management and Monitoring Programming Examples +--- + +One example demonstrates the use of an MBean server to manage and monitor a node in a distributed system, and the other example acts as the managed node. + +## JMX Manager Node Example + +``` pre +package quickstart; + +import java.util.Set; +import javax.management.ObjectName; +import org.apache.geode.cache.Cache; +import org.apache.geode.cache.CacheFactory; +import org.apache.geode.distributed.DistributedMember; +import org.apache.geode.management.ManagementService; +import org.apache.geode.management.RegionMXBean; + +/** + * Example of a JMX Manager Node. It first creates a cache. + * Then for each member of Distributed System it retrieves RegionMXBean and uses its data. + * + * @author GemStone Systems, Inc. + * + * @since 7.0 + */ +public class ManagerNode { + + /** + * @param args + */ + public static final String EXAMPLE_REGION_NAME = "exampleRegion"; + + public static void main(String[] args) throws Exception { + + //Set waiting period for federation in milliseconds + final int JMX_WAIT_PERIOD_FOR_FEDERATION_UPDATE = 2500; + + + System.out + .println("Connecting to the distributed system and creating the cache."); + + // Create the cache which causes the cache-xml-file to be parsed + Cache cache = new CacheFactory() + .set("name", "ManagerNode") + .set("statistic-sampling-enabled", "true") + .set("jmx-manager-start","true") + .set("jmx-manager","true") + .set("jmx-manager-port","2576") + .set("cache-xml-file", "xml/Managed-node.xml").create(); + + // Retrieve service + System.out.println("Retrieving service ..."); + + //ManagementService.getManagementService() will create a service in case not already created + //ManagementService.getExistingManagementService() will return existing one, otherwise null. + ManagementService service = ManagementService.getManagementService(cache); + System.out.println("Retrieved service "); + + + //Retrieve distributed system members + System.out.println("Retrieving distributed members ..."); + Set<DistributedMember> dsMembers = cache.getDistributedSystem() + .getAllOtherMembers(); + System.out.println("Retrieved " +dsMembers.size()+ " distributed members "); + + + //Sleep is needed for federation data to get published + System.out.println("Retrieving RegionMXBean for members in DS ..."); + + try { + Thread.sleep(JMX_WAIT_PERIOD_FOR_FEDERATION_UPDATE); + } catch (Exception e) { + + } + + for (DistributedMember dsMember : dsMembers) { + System.out.println("Retrieved RegionMXBean for member "+ dsMember.getId()); + + //Get region member bean name using region path. + ObjectName regionMBeanName = service.getRegionMBeanName(dsMember, "/" + + EXAMPLE_REGION_NAME); + System.out.println("regionMBeanName " + regionMBeanName); + + //Get Region MBean Proxy + RegionMXBean regionMXBean = service.getMBeanInstance(regionMBeanName, + RegionMXBean.class); + + // Validate regionMXbean + if(regionMXBean != null){ + System.out.println("Entry count in " + EXAMPLE_REGION_NAME + " is =" + regionMXBean.getEntryCount()); + }else{ + System.out.println("Retrieved RegionMXBean is null."); + } + } + + // Close the cache and disconnect from GemFire distributed system + System.out.println("Closing the cache and disconnecting."); + cache.close(); + + + //Complete Managed Node by pressing enter in managed node console + System.out.println(""); + System.out.println("Press enter in Managed Node"); + + } +} +``` + +## GemFire Managed Node Example + +``` pre +package quickstart; + +import java.io.BufferedReader; +import java.io.InputStreamReader; + +import org.apache.geode.cache.Cache; +import org.apache.geode.cache.CacheFactory; +import org.apache.geode.cache.Region; +import org.apache.geode.management.ManagementService; +import org.apache.geode.management.RegionMXBean; + +/** + * In this example of Managed Node. + * 1. It creates a cache and adds entries in it. + * 2. From ManagementService it retrieves service. + * 3. From service gets Region Mbean. + * + * + * @author GemStone Systems, Inc. + * + * @since 7.0 + */ + +public class ManagedNode { + + /** + * @param args + */ + public static final String EXAMPLE_REGION_NAME = "exampleRegion"; + + public static void main(String[] args) throws Exception { + + //Set waiting period for federation in milliseconds + final int JMX_WAIT_PERIOD_FOR_FEDERATION_UPDATE = 2500; + + System.out + .println("Connecting to the distributed system and creating the cache."); + + // Create the cache which causes the cache-xml-file to be parsed + System.out.println("Creating cache using Mbean.xml"); + Cache cache = new CacheFactory().set("name", "ManagedNode") + .set("log-level", "fine") + .set("statistic-sampling-enabled", "true") + .set("cache-xml-file", "xml/Managed-node.xml").create(); + + System.out.println("Created cache using Mbean.xml"); + + // Create one sample region + Region<Object, Object> region = cache.getRegion("/" + EXAMPLE_REGION_NAME); + + // Add some entries to region + if (region != null) { + System.out.println("Adding key and value to region in member " + + cache.getDistributedSystem().getDistributedMember().getId()); + for (int count = 0; count < 10; count++) { + String key = "key" + count; + String value = "value" + count; + region.put(key, value); + } + } + + System.out.println("Retrieving service using cache ..."); + + // Retrieve service in this node + ManagementService service = ManagementService + .getExistingManagementService(cache); + System.out.println("Retrieved service."); + + System.out.println("Retrieving RegionMXBean for " + EXAMPLE_REGION_NAME); + + try{ + Thread.sleep(JMX_WAIT_PERIOD_FOR_FEDERATION_UPDATE); + }catch(Exception e){ + + } + + RegionMXBean regionBean = service.getLocalRegionMBean("/" + + EXAMPLE_REGION_NAME); + + + + if (regionBean != null) { + System.out.println("Retrieved RegionMXBean for " + EXAMPLE_REGION_NAME); + System.out.println("Entry count for " + EXAMPLE_REGION_NAME + " is=" + + regionBean.getEntryCount()); + + System.out.println("Start Manager Node and wait till it completes"); + + } else { + System.out.println("Could not retriev RegionMXBean for " + + EXAMPLE_REGION_NAME); + } + + // Wait till user input, after manager node completes + BufferedReader bufferedReader = new BufferedReader(new InputStreamReader( + System.in)); + bufferedReader.readLine(); + + // Close the cache and disconnect + System.out.println("Closing the cache and disconnecting."); + cache.close(); + } + +} +``` + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/monitor_tune/cache_consistency.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/cache_consistency.html.md.erb b/geode-docs/managing/monitor_tune/cache_consistency.html.md.erb new file mode 100644 index 0000000..09f868b --- /dev/null +++ b/geode-docs/managing/monitor_tune/cache_consistency.html.md.erb @@ -0,0 +1,63 @@ +--- +title: Maintaining Cache Consistency +--- + +Maintaining data consistency between caches in a distributed Geode system is vital for ensuring its functional integrity and preventing data loss. + +## <a id="cache_const__section_lf3_lvn_nr" class="no-quick-link"></a>General Guidelines + +**Before Restarting a Region with a Disk Store, Consider the State of the Entire Region** + +**Note:** +If you revoke a memberâs disk store, do not restart that member with its disk storesâin isolationâat a later time. + +Geode stores information about your persisted data and prevents you from starting a member with a revoked disk store in the running system. But Geode cannot stop you from starting a revoked member in isolation, and running with its revoked data. This is an unlikely situation, but it is possible to do: + +1. Members A and B are running, both storing Region data to disk. +2. Member A goes down. +3. Member B goes down. +4. At this point, Member B has the most recent disk data. +5. Member B is not usable. Perhaps its host machine is down or cut off temporarily. +6. To get the system up and running, you start Member A, and use the command line tool to revoke Member Bâs status as member with the most recent data. The system loads Member Aâs data and you run forward with that. +7. Member A is stopped. +8. At this point, both Member A and Member B have information in their disk files indicating they are the gold copy members. +9. If you start Member B, it will load its data from disk. +10. When you start Member A, the system will recognize the incompatible state and report an exception, but by this point, you have good data in both files, with no way to combine them. + +**Understand Cache Transactions** + +Understanding the operation of Geode transactions can help you minimize situations where the cache could get out of sync. + +Transactions do not work in distributed regions with global scope. + +Transactions provide consistency within one cache, but the distribution of results to other members is not as consistent. + +Multiple transactions in a cache can create inconsistencies because of read committed isolation. Since multiple threads cannot participate in a transaction, most applications will be running multiple transactions. + +An in-place change to directly alter a keyâs value without doing a put can result in cache inconsistencies. With transactions, it creates additional difficulties because it breaks read committed isolation. If at all possible, use copy-on-read instead. + +In distributed-no-ack scope, two conflicting transactions in different members can commit simultaneously, overwriting each other as the changes are distributed. + +If a cache writer exists during a transaction, then each transaction write operation triggers a cache writerâs related call. Regardless of the regionâs scope, a transaction commit can invoke a cache writer only in the local cache and not in the remote caches. + +A region in a cache with transactions may not stay in sync with a region of the same name in another cache without transactions. + +Two applications running the same sequence of operations in their transactions may get different results. This could occur because operations happening outside a transaction in one of the members can overwrite the transaction, even in the process of committing. This could also occur if the results of a large transaction exceed the machineâs memory or the capacity of Geode. Those limits can vary by machine, so the two members may not be in sync. + +## <a id="cache_const__section_qxx_kvn_nr" class="no-quick-link"></a>Guidelines for Multi-Site Deployments + +**Optimize socket-buffer-size** + +In a multi-site installation using gateways, if the link between sites is not tuned for optimum throughput, it could cause messages to back up in the cache queues. If a queue overflows because of inadequate buffer sizes, it will become out of sync with the sender and the receiver will be unaware of the condition. You can configure the send-receive buffer sizes of the TCP/IP connections used for data transmissions by changing the socket-buffer-size attribute of the gateway-sender and gateway-receiver elements in the `cache.xml` file. Set the buffer size by determining the link bandwidth and then using ping to measure the round-trip time. + +When optimizing socket-buffer sizes, use the same value for both gateway senders and gateway receivers. + +**Prevent Primary and Secondary Gateway Senders from Going Offline** + +In a multi-site installation, if the primary gateway server goes offline, a secondary gateway sender must take over primary responsibilities as the failover system. The existing secondary gateway sender detects that the primary gateway sender has gone offline, and a secondary one becomes the new primary. Because the queue is distributed, its contents are available to all gateway senders. So, when a secondary gateway sender becomes primary, it is able to start processing the queue where the previous primary left off with no loss of data. + +If both the primary gateway sender and all its secondary senders go offline and messages are in their queues, data loss could occur, because there is no failover system. + +**Verify That isOriginRemote Is Set to False** + +The isOriginRemote flag for a server or a multi-site gateway is set to false by default, which ensures that updates are distributed to other members. Setting its value to true in the server or the receiving gateway member applies updates to that member only, so updates are not distributed to peer members. http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/monitor_tune/chapter_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/chapter_overview.html.md.erb b/geode-docs/managing/monitor_tune/chapter_overview.html.md.erb new file mode 100644 index 0000000..da69f00 --- /dev/null +++ b/geode-docs/managing/monitor_tune/chapter_overview.html.md.erb @@ -0,0 +1,43 @@ +--- +title: Performance Tuning and Configuration +--- + +A collection of tools and controls allow you to monitor and adjust Apache Geode performance. + +- **[Improving Geode Performance on vSphere](../../managing/monitor_tune/gemfire_performance_on_vsphere.html)** + + This topic provides guidelines for tuning vSphere virtualized environments that host Apache Geode deployments. + +- **[Performance Controls](../../managing/monitor_tune/performance_controls.html)** + + This topic provides tuning suggestions of particular interest to developers, primarily programming techniques and cache configuration. + +- **[System Member Performance](../../managing/monitor_tune/system_member_performance.html)** + + You can modify some configuration parameters to improve system member performance. + +- **[Slow Receivers with TCP/IP](../../managing/monitor_tune/slow_receivers.html)** + + You have several options for preventing situations that can cause slow receivers of data distributions. The slow receiver options control only peer-to-peer communication using TCP/IP. This discussion does not apply to client/server or multi-site communication, or to communication using the UDP unicast or multicast protocols. + +- **[Slow distributed-ack Messages](../../managing/monitor_tune/slow_messages.html)** + + In systems with distributed-ack regions, a sudden large number of distributed-no-ack operations can cause distributed-ack operations to take a long time to complete. + +- **[Socket Communication](../../managing/monitor_tune/socket_communication.html)** + + Geode processes communicate using TCP/IP and UDP unicast and multicast protocols. In all cases, communication uses sockets that you can tune to optimize performance. + +- **[UDP Communication](../../managing/monitor_tune/udp_communication.html)** + + You can make configuration adjustments to improve multicast and unicast UDP performance of peer-to-peer communication. + +- **[Multicast Communication](../../managing/monitor_tune/multicast_communication.html)** + + You can make configuration adjustments to improve the UDP multicast performance of peer-to-peer communication in your Geode system. + +- **[Maintaining Cache Consistency](../../managing/monitor_tune/cache_consistency.html)** + + Maintaining data consistency between caches in a distributed Geode system is vital for ensuring its functional integrity and preventing data loss. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere.html.md.erb b/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere.html.md.erb new file mode 100644 index 0000000..249a7dd --- /dev/null +++ b/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere.html.md.erb @@ -0,0 +1,47 @@ +--- +title: Improving Geode Performance on vSphere +--- + +This topic provides guidelines for tuning vSphere virtualized environments that host Apache Geode deployments. + +Without tuning, Geode can suffer a performance drop in virtual environments, including the VMware vSphere virtual platform. + +You can expect to see significant performance degradation when running Geode on vSphere versus running Geode on dedicated hardware. + +- **[Operating System Guidelines](gemfire_performance_on_vsphere_guidelines.html#topic_F48990A6A37144988D49E132E17E117C)** + + Use the latest supported version of the guest OS, and use Java large paging. + +- **[NUMA, CPU, and BIOS Settings](gemfire_performance_on_vsphere_guidelines.html#topic_D8393B1A75364E46B0F959F0DE820E9E)** + + This section provides VMware- recommended NUMA, CPU, and BIOS settings for your hardware and virtual machines. + +- **[Physical and Virtual NIC Settings](gemfire_performance_on_vsphere_guidelines.html#topic_7A5F1EAD7A6C4E21BB1FF7CF3B625BC5)** + + These guidelines help you reduce latency. + +- **[VMware vSphere vMotion and DRS Cluster Usage](gemfire_performance_on_vsphere_guidelines.html#topic_E6EB8AB6CCEF435A98B48B867FE9BFEB)** + + This topic discusses use limitations of vSphere vMotion, including the use of it with DRS. + +- **[Placement and Organization of Virtual Machines](gemfire_performance_on_vsphere_guidelines.html#topic_E53BBF3D09A54953B02DCE2BD00D51E0)** + + This section provides guidelines on JVM instances and placement of redundant copies of cached data. + +- **[Virtual Machine Memory Reservation](gemfire_performance_on_vsphere_guidelines.html#topic_567308E9DE07406BB5BF420BE77B6558)** + + This section provides guidelines for sizing and setting memory. + +- **[vSphere High Availability and Apache Geode](gemfire_performance_on_vsphere_guidelines.html#topic_424B940584044CF6A685E86802548A27)** + + On Apache Geode virtual machines, disable vSphere High Availability (HA). + +- **[Storage Guidelines](gemfire_performance_on_vsphere_guidelines.html#topic_913B15841C4249A68697F3D91281A645)** + + This section provides storage guidelines for persistence files, binaries, logs, and more. + +- **[Additional Resources](gemfire_performance_on_vsphere_guidelines.html#topic_628F038FD4954E56BF4192F17FD3D119)** + + VMware provides additional resources for optimizing vSphere, Java applications, and Apache Geode. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere_guidelines.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere_guidelines.html.md.erb b/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere_guidelines.html.md.erb new file mode 100644 index 0000000..d8b7248 --- /dev/null +++ b/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere_guidelines.html.md.erb @@ -0,0 +1,119 @@ +--- +title: Operating System Guidelines +--- +<a id="topic_F48990A6A37144988D49E132E17E117C"></a> + + +Use the latest supported version of the guest OS, and use Java large paging. + +- **Use the latest supported version of the guest operating system**. This guideline is probably the most important. Upgrade the guest OS to a recent version supported by Geode. For example, for RHEL, use at least version 7.0 or for SLES, use at least 11.0. For Windows, use Windows Server 2012. For RedHat Linux users, it is particularly beneficial to use RHEL 7 since there are specific enhancements in the RHEL 7 release that improve virtualized latency sensitive workloads. +- **Use Java large paging in guest OS**. Configure Java on the guest OS to use large pages. Add the following command line option when launching Java: + + ``` pre + -XX:+UseLargePages + ``` + +## <a id="topic_D8393B1A75364E46B0F959F0DE820E9E" class="no-quick-link"></a>NUMA, CPU, and BIOS Settings + +This section provides VMware- recommended NUMA, CPU, and BIOS settings for your hardware and virtual machines. + +- Always enable hyper-threading, and do not overcommit CPU. +- For most production Apache Geode servers, always use virtual machines with at least two vCPUs . +- Apply non-uniform memory access (NUMA) locality by sizing virtual machines to fit within the NUMA node. +- VMware recommends the following BIOS settings: + - **BIOS Power Management Mode:** Maximum Performance. + - **CPU Power and Performance Management Mode:** Maximum Performance. + - **Processor Settings:**Turbo Mode enabled. + - **Processor Settings:**C States disabled. + +**Note:** +Settings may vary slightly depending on your hardware make and model. Use the settings above or equivalents as needed. + +## <a id="topic_7A5F1EAD7A6C4E21BB1FF7CF3B625BC5" class="no-quick-link"></a>Physical and Virtual NIC Settings + +These guidelines help you reduce latency. + +- **Physical NIC:** VMware recommends that you disable interrupt coalescing on the physical NIC of your ESXi host by using the following command: + + ``` pre + ethtool -C vmnicX rx-usecs 0 rx-frames 1 rx-usecs-irq 0 rx-frames-irq 0 + ``` + + where `vmnicX` is the physical NIC as reported by the ESXi command: + + ``` pre + esxcli network nic list + ``` + + You can verify that your settings have taken effect by issuing the command: + + ``` pre + ethtool -C vmnicX + ``` + + If you restart the ESXi host, the above configuration must be reapplied. + + **Note:** + Disabling interrupt coalescing can reduce latency in virtual machines; however, it can impact performance and cause higher CPU utilization. It can also defeat the benefits of Large Receive Offloads (LRO) because some physical NICs (such as Intel 10GbE NICs) automatically disable LRO when interrupt coalescing is disabled. See [http://kb.vmware.com/kb/1027511](http://kb.vmware.com/kb/1027511) for more details. + +- **Virtual NIC:** Use the following guidelines when configuring your virtual NICs: + - Use VMXNET3 virtual NICs for your latency-sensitive or otherwise performance-critical virtual machines. See [http://kb.vmware.com/kb/1001805](http://kb.vmware.com/kb/1001805) for details on selecting the appropriate type of virtual NIC for your virtual machine. + - VMXNET3 supports adaptive interrupt coalescing that can help drive high throughput to virtual machines that have multiple vCPUs with parallelized workloads (multiple threads), while minimizing latency of virtual interrupt delivery. However, if your workload is extremely sensitive to latency, VMware recommends that you disable virtual interrupt coalescing for your virtual NICs. You can do this programmatically via API or by editing your virtual machine's .vmx configuration file. Refer to your vSphere API Reference or VMware ESXi documentation for specific instructions. + +## <a id="topic_E6EB8AB6CCEF435A98B48B867FE9BFEB" class="no-quick-link"></a>VMware vSphere vMotion and DRS Cluster Usage + +This topic discusses use limitations of vSphere vMotion, including the use of it with DRS. + +- When you first commission the data management system, place VMware vSphere Distributed Resource Scheduler⢠(DRS) in manual mode to prevent an automatic VMware vSphere vMotion® operation that can affect response times. +- Reduce or eliminate the use of vMotion to migrate Geode virtual machines when they are under heavy load. +- Do not allow vMotion migrations with Apache Geode locator processes, as the latency introduced to this process can cause other members of the Apache Geode servers to falsely suspect that other members are dead. +- Use dedicated Apache Geode vSphere DRS clusters. This is especially important when you consider that the physical NIC and virtual NIC are specifically tuned to disable Interrupt Coalescing on every NIC of an ESXi host in the cluster. This type of tuning benefits Geode workloads, but it can hurt other non-Apache Geode workloads that are memory throughput-bound as opposed to latency sensitive as in the case of Apache Geode workloads. +- If using a dedicated vSphere DRS cluster is not an option, and Apache Geode must run in a shared DRS cluster, make sure that DRS rules are set up not to perform vMotion migrations on Geode virtual machines. +- If you must use vMotion for migration, VMware recommends that all vMotion migration activity of Apache Geode members occurs over 10GbE, during periods of low activity and scheduled maintenance windows. + +## <a id="topic_E53BBF3D09A54953B02DCE2BD00D51E0" class="no-quick-link"></a>Placement and Organization of Virtual Machines + +This section provides guidelines on JVM instances and placement of redundant copies of cached data. + +- Have one JVM instance per virtual machine. +- Increasing the heap space to service the demand for more data is better than installing a second instance of a JVM on a single virtual machine. If increasing the JVM heap size is not an option, consider placing the second JVM on a separate newly created virtual machine, thus promoting more effective horizontal scalability. As you increase the number of Apache Geode servers, also increase the number of virtual machines to maintain a 1:1:1 ratio among the Apache Geode server, the JVM, and the virtual machines. +- Size for a minimum of four vCPU virtual machines with one Apache Geode server running in one JVM instance. This allows ample CPU cycles for the garbage collector, and the rest for user transactions. +- Because Apache Geode can place redundant copies of cached data on any virtual machine, it is possible to inadvertently place two redundant data copies on the same ESX/ESXi host. This is not optimal if a host fails. To create a more robust configuration, use VM1-to-VM2 anti-affinity rules, to indicate to vSphere that VM1 and VM2 can never be placed on the same host because they hold redundant data copies. + +## <a id="topic_567308E9DE07406BB5BF420BE77B6558" class="no-quick-link"></a>Virtual Machine Memory Reservation + +This section provides guidelines for sizing and setting memory. + +- Set memory reservation at the virtual machine level so that ESXi provides and locks down the needed physical memory upon virtual machine startup. Once allocated, ESXi does not allow the memory to be taken away. +- Do not overcommit memory for Geode hosts. +- When sizing memory for a Geode server within one JVM on one virtual machine, the total reserved memory for the virtual machine should not exceed what is available within one NUMA node for optimal performance. + +## <a id="topic_424B940584044CF6A685E86802548A27" class="no-quick-link"></a>vSphere High Availability and Apache Geode + +On Apache Geode virtual machines, disable vSphere High Availability (HA). + +If you are using a dedicated Apache Geode DRS cluster, then you can disable HA across the cluster. However, if you are using a shared cluster, exclude Geode virtual machines from vSphere HA. + +Additionally, to support high availability, you can also set up anti-affinity rules between the Apache Geode virtual machines to prevent two Apache Geode servers from running on the same ESXi host within the same DRS cluster. + +## <a id="topic_913B15841C4249A68697F3D91281A645" class="no-quick-link"></a>Storage Guidelines + +This section provides storage guidelines for persistence files, binaries, logs, and more. + +- Use the PVSCSI driver for I/O intensive Apache Geode workloads. +- Align disk partitions at the VMFS and guest operating system levels. +- Provision VMDK files as eagerzeroedthick to avoid lazy zeroing for Apache Geode members. +- Use separate VMDKs for Apache Geode persistence files, binaries, and logs. +- Map a dedicated LUN to each VMDK. +- For Linux virtual machines, use NOOP scheduling as the I/O scheduler instead of Completely Fair Queuing (CFQ). Starting with the Linux kernel 2.6, CFQ is the default I/O scheduler in many Linux distributions. See [http://kb.vmware.com/kb/2011861](http://kb.vmware.com/kb/2011861) for more information. + +## <a id="topic_628F038FD4954E56BF4192F17FD3D119" class="no-quick-link"></a>Additional Resources + +VMware provides additional resources for optimizing vSphere, Java applications, and Apache Geode. + +- "Performance Best Practices for VMware vSphere 5.0" - [http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf](http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf) +- "Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere Virtual Machines" - [http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf](http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf) +- "Enterprise Java Applications on VMware - Best Practices Guide" - [http://www.vmware.com/resources/techresources/1087](http://www.vmware.com/resources/techresources/1087) +- "High Performance Data with VMware Pivotal⢠GemFire® Best Practices Guide" - [https://www.vmware.com/files/pdf/techpaper/vmw-vfabric-gemFire-best-practices-guide.pdf](https://www.vmware.com/files/pdf/techpaper/vmw-vfabric-gemFire-best-practices-guide.pdf) + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/monitor_tune/multicast_communication.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication.html.md.erb new file mode 100644 index 0000000..3574cc7 --- /dev/null +++ b/geode-docs/managing/monitor_tune/multicast_communication.html.md.erb @@ -0,0 +1,29 @@ +--- +title: Multicast Communication +--- + +You can make configuration adjustments to improve the UDP multicast performance of peer-to-peer communication in your Geode system. + +Before you begin, you should understand Geode [Basic Configuration and Programming](../../basic_config/book_intro.html). See also the general communication tuning and UDP tuning covered in [Socket Communication](socket_communication.html) and [UDP Communication](udp_communication.html#udp_comm). + +- **[Provisioning Bandwidth for Multicast](../../managing/monitor_tune/multicast_communication_provisioning_bandwidth.html)** + + Multicast installations require more planning and configuration than TCP installations. With IP multicast, you gain scalability but lose the administrative convenience of TCP. + +- **[Testing Multicast Speed Limits](../../managing/monitor_tune/multicast_communication_testing_multicast_speed_limits.html)** + + TCP automatically adjusts its speed to the capability of the processes using it and enforces bandwidth sharing so that every process gets a turn. With multicast, you must determine and explicitly set those limits. + +- **[Configuring Multicast Speed Limits](../../managing/monitor_tune/multicast_communication_configuring_speed_limits.html)** + + After you determine the maximum transmission rate, configure and tune your production system. + +- **[Run-time Considerations for Multicast](../../managing/monitor_tune/multicast_communication_runtime_considerations.html)** + + When you use multicast for messaging and data distribution, you need to understand how the health monitoring setting works and how to control memory use. + +- **[Troubleshooting the Multicast Tuning Process](../../managing/monitor_tune/multicast_communication_troubleshooting.html)** + + Several problems may arise during the initial testing and tuning process for multicasting. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/monitor_tune/multicast_communication_configuring_speed_limits.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication_configuring_speed_limits.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication_configuring_speed_limits.html.md.erb new file mode 100644 index 0000000..8ef2894 --- /dev/null +++ b/geode-docs/managing/monitor_tune/multicast_communication_configuring_speed_limits.html.md.erb @@ -0,0 +1,34 @@ +--- +title: Configuring Multicast Speed Limits +--- + +After you determine the maximum transmission rate, configure and tune your production system. + +<a id="multicast__section_8E225FC6829946C287552BC7996F2765"></a> +For best performance, the producer and the consumers should run on different machines and each process should have at least one CPU dedicated to it. The following is a list of configuration changes that can improve multicast performance. Check with your system administrator about changing any of the limits discussed here. + +- Increase the default datagram size for systems running Microsoft Windows from 1024 bytes to a value that matches your networkâs maximum transmission unit (MTU), which is typically 1500 bytes. The higher setting should improve the systemâs network performance. +- Distribution statistics for stack time probes are disabled by default to increase multicast performance. To reduce multicast speed, you can enable time statistics by setting the gemfire.`enable-time-statistics` property to true. + + This enables time statistics for a Java application: + + ``` pre + -Dgemfire.enable-time-statistics=true + ``` + + The time statistics properties are passed to the cache server on the `gfsh` the command line: + + ``` pre + gfsh>start server --name=server_name --enable-time-statistics=true + ``` + +- Monitor the members that receive data for signs of data loss. A few data loss messages can happen normally during region creation. Multicast retransmit requests and unicast retransmits can also be monitored to detect data loss. Even when you see data loss, the cause of the problem may have nothing to do with the network. However, if it happens constantly then you should try testing the flow control rate again +- If necessary, reconfigure all the `gemfire.properties` files and repeat with lower flow control maximum credits until you find the maximum useful rate for your installation. +- Slow system performance might be helped by reducing how far your multicast messaging goes in your network. +- Reduce multicast latency by disabling batching. By default, Geode uses batching for operations when the regionâs scope is distributed-no-ack. Set the `disableBatching` property to true on the application or when starting a cache server process through the `gfsh` command line: + + ``` pre + gfsh>start server --name=server_name --J=-Dp2p.disableBatching=true + ``` + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/managing/monitor_tune/multicast_communication_provisioning_bandwidth.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication_provisioning_bandwidth.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication_provisioning_bandwidth.html.md.erb new file mode 100644 index 0000000..d236823 --- /dev/null +++ b/geode-docs/managing/monitor_tune/multicast_communication_provisioning_bandwidth.html.md.erb @@ -0,0 +1,43 @@ +--- +title: Provisioning Bandwidth for Multicast +--- + +Multicast installations require more planning and configuration than TCP installations. With IP multicast, you gain scalability but lose the administrative convenience of TCP. + +<a id="multicast__section_B7DA88707CBF4713A1E287CAA9A80EB9"></a> +When you install an application that runs over TCP, the network is almost always set up for TCP and other applications are already using it. When you install an application to run over IP multicast it may be the first multicast application on the network. + +Multicast is very dependent on the environment in which it runs. Its operation is affected by the network hardware, the network software, the machines, which Geode processes run on which machines, and whether there are any competing applications. You could find that your site has connectivity in TCP but not in multicast because some switches and network cards do not support multicast. Your network could have latent problems that you would never see otherwise. To successfully implement a distributed Geode system using multicast requires the cooperation of both system and network administrators. + +**Bounded Operation Over Multicast** + +Group rate control is required for Geode systems to maintain cache coherence. If your application delivers the same data to a group of members, your system tuning effort needs to focus on the slow receivers. + +If some of your members have trouble keeping up with the incoming data, the other members in the group may be impacted. At best, slow receivers cause the producer to use buffering, adding latency for the slow receiver and perhaps for all of them. In the worst case, throughput for the group can stop entirely while the producerâs CPU, memory and network bandwidth are dedicated to serving the slow receivers. + +To address this issue, you can implement a bounded operation policy, which sets boundaries for the producerâs operation. The appropriate rate limits are determined through tuning and testing to allow the fastest operation possible while minimizing data loss and latency in the group of consumers. This policy is suited to applications such as financial market data, where high throughput, reliable delivery and network stability are required. With the boundaries set correctly, your producerâs traffic cannot cause a network outage. + +Multicast protocols typically have a flow control protocol built into them to keep processes from being overrun. The Geode flow control protocol uses the mcast-flow-control property to set producer and consumer boundaries for multicast flow operations. The property provides these three configuration settings: + +<table> +<colgroup> +<col width="50%" /> +<col width="50%" /> +</colgroup> +<tbody> +<tr class="odd"> +<td><p><code class="ph codeph">byteAllowance</code></p></td> +<td><p>Number of bytes that can be sent without a recharge.</p></td> +</tr> +<tr class="even"> +<td><p><code class="ph codeph">rechargeThreshold</code></p></td> +<td><p>Tells consumers how low the producerâs initial to remaining allowance ratio should be before sending a recharge.</p></td> +</tr> +<tr class="odd"> +<td><code class="ph codeph">rechargeBlockMs</code></td> +<td><p>Tells the producer how long to wait for a recharge before requesting one.</p></td> +</tr> +</tbody> +</table> + +
