http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/cache_consistency.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/cache_consistency.html.md.erb b/geode-docs/managing/monitor_tune/cache_consistency.html.md.erb deleted file mode 100644 index adc10c9..0000000 --- a/geode-docs/managing/monitor_tune/cache_consistency.html.md.erb +++ /dev/null @@ -1,80 +0,0 @@ ---- -title: Maintaining Cache Consistency ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -Maintaining data consistency between caches in a distributed Geode system is vital for ensuring its functional integrity and preventing data loss. - -## <a id="cache_const__section_lf3_lvn_nr" class="no-quick-link"></a>General Guidelines - -**Before Restarting a Region with a Disk Store, Consider the State of the Entire Region** - -**Note:** -If you revoke a memberâs disk store, do not restart that member with its disk storesâin isolationâat a later time. - -Geode stores information about your persisted data and prevents you from starting a member with a revoked disk store in the running system. But Geode cannot stop you from starting a revoked member in isolation, and running with its revoked data. This is an unlikely situation, but it is possible to do: - -1. Members A and B are running, both storing Region data to disk. -2. Member A goes down. -3. Member B goes down. -4. At this point, Member B has the most recent disk data. -5. Member B is not usable. Perhaps its host machine is down or cut off temporarily. -6. To get the system up and running, you start Member A, and use the command line tool to revoke Member Bâs status as member with the most recent data. The system loads Member Aâs data and you run forward with that. -7. Member A is stopped. -8. At this point, both Member A and Member B have information in their disk files indicating they are the gold copy members. -9. If you start Member B, it will load its data from disk. -10. When you start Member A, the system will recognize the incompatible state and report an exception, but by this point, you have good data in both files, with no way to combine them. - -**Understand Cache Transactions** - -Understanding the operation of Geode transactions can help you minimize situations where the cache could get out of sync. - -Transactions do not work in distributed regions with global scope. - -Transactions provide consistency within one cache, but the distribution of results to other members is not as consistent. - -Multiple transactions in a cache can create inconsistencies because of read committed isolation. Since multiple threads cannot participate in a transaction, most applications will be running multiple transactions. - -An in-place change to directly alter a keyâs value without doing a put can result in cache inconsistencies. With transactions, it creates additional difficulties because it breaks read committed isolation. If at all possible, use copy-on-read instead. - -In distributed-no-ack scope, two conflicting transactions in different members can commit simultaneously, overwriting each other as the changes are distributed. - -If a cache writer exists during a transaction, then each transaction write operation triggers a cache writerâs related call. Regardless of the regionâs scope, a transaction commit can invoke a cache writer only in the local cache and not in the remote caches. - -A region in a cache with transactions may not stay in sync with a region of the same name in another cache without transactions. - -Two applications running the same sequence of operations in their transactions may get different results. This could occur because operations happening outside a transaction in one of the members can overwrite the transaction, even in the process of committing. This could also occur if the results of a large transaction exceed the machineâs memory or the capacity of Geode. Those limits can vary by machine, so the two members may not be in sync. - -## <a id="cache_const__section_qxx_kvn_nr" class="no-quick-link"></a>Guidelines for Multi-Site Deployments - -**Optimize socket-buffer-size** - -In a multi-site installation using gateways, if the link between sites is not tuned for optimum throughput, it could cause messages to back up in the cache queues. If a queue overflows because of inadequate buffer sizes, it will become out of sync with the sender and the receiver will be unaware of the condition. You can configure the send-receive buffer sizes of the TCP/IP connections used for data transmissions by changing the socket-buffer-size attribute of the gateway-sender and gateway-receiver elements in the `cache.xml` file. Set the buffer size by determining the link bandwidth and then using ping to measure the round-trip time. - -When optimizing socket-buffer sizes, use the same value for both gateway senders and gateway receivers. - -**Prevent Primary and Secondary Gateway Senders from Going Offline** - -In a multi-site installation, if the primary gateway server goes offline, a secondary gateway sender must take over primary responsibilities as the failover system. The existing secondary gateway sender detects that the primary gateway sender has gone offline, and a secondary one becomes the new primary. Because the queue is distributed, its contents are available to all gateway senders. So, when a secondary gateway sender becomes primary, it is able to start processing the queue where the previous primary left off with no loss of data. - -If both the primary gateway sender and all its secondary senders go offline and messages are in their queues, data loss could occur, because there is no failover system. - -**Verify That isOriginRemote Is Set to False** - -The isOriginRemote flag for a server or a multi-site gateway is set to false by default, which ensures that updates are distributed to other members. Setting its value to true in the server or the receiving gateway member applies updates to that member only, so updates are not distributed to peer members.
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/chapter_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/chapter_overview.html.md.erb b/geode-docs/managing/monitor_tune/chapter_overview.html.md.erb deleted file mode 100644 index db09945..0000000 --- a/geode-docs/managing/monitor_tune/chapter_overview.html.md.erb +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: Performance Tuning and Configuration ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -A collection of tools and controls allow you to monitor and adjust Apache Geode performance. - -- **[Improving Geode Performance on vSphere](../../managing/monitor_tune/gemfire_performance_on_vsphere.html)** - - This topic provides guidelines for tuning vSphere virtualized environments that host Apache Geode deployments. - -- **[Performance Controls](../../managing/monitor_tune/performance_controls.html)** - - This topic provides tuning suggestions of particular interest to developers, primarily programming techniques and cache configuration. - -- **[System Member Performance](../../managing/monitor_tune/system_member_performance.html)** - - You can modify some configuration parameters to improve system member performance. - -- **[Slow Receivers with TCP/IP](../../managing/monitor_tune/slow_receivers.html)** - - You have several options for preventing situations that can cause slow receivers of data distributions. The slow receiver options control only peer-to-peer communication using TCP/IP. This discussion does not apply to client/server or multi-site communication, or to communication using the UDP unicast or multicast protocols. - -- **[Slow distributed-ack Messages](../../managing/monitor_tune/slow_messages.html)** - - In systems with distributed-ack regions, a sudden large number of distributed-no-ack operations can cause distributed-ack operations to take a long time to complete. - -- **[Socket Communication](../../managing/monitor_tune/socket_communication.html)** - - Geode processes communicate using TCP/IP and UDP unicast and multicast protocols. In all cases, communication uses sockets that you can tune to optimize performance. - -- **[UDP Communication](../../managing/monitor_tune/udp_communication.html)** - - You can make configuration adjustments to improve multicast and unicast UDP performance of peer-to-peer communication. - -- **[Multicast Communication](../../managing/monitor_tune/multicast_communication.html)** - - You can make configuration adjustments to improve the UDP multicast performance of peer-to-peer communication in your Geode system. - -- **[Maintaining Cache Consistency](../../managing/monitor_tune/cache_consistency.html)** - - Maintaining data consistency between caches in a distributed Geode system is vital for ensuring its functional integrity and preventing data loss. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere.html.md.erb b/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere.html.md.erb deleted file mode 100644 index 2be5502..0000000 --- a/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere.html.md.erb +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: Improving Geode Performance on vSphere ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -This topic provides guidelines for tuning vSphere virtualized environments that host Apache Geode deployments. - -Without tuning, Geode can suffer a performance drop in virtual environments, including the VMware vSphere virtual platform. - -You can expect to see significant performance degradation when running Geode on vSphere versus running Geode on dedicated hardware. - -- **[Operating System Guidelines](gemfire_performance_on_vsphere_guidelines.html#topic_F48990A6A37144988D49E132E17E117C)** - - Use the latest supported version of the guest OS, and use Java large paging. - -- **[NUMA, CPU, and BIOS Settings](gemfire_performance_on_vsphere_guidelines.html#topic_D8393B1A75364E46B0F959F0DE820E9E)** - - This section provides VMware- recommended NUMA, CPU, and BIOS settings for your hardware and virtual machines. - -- **[Physical and Virtual NIC Settings](gemfire_performance_on_vsphere_guidelines.html#topic_7A5F1EAD7A6C4E21BB1FF7CF3B625BC5)** - - These guidelines help you reduce latency. - -- **[VMware vSphere vMotion and DRS Cluster Usage](gemfire_performance_on_vsphere_guidelines.html#topic_E6EB8AB6CCEF435A98B48B867FE9BFEB)** - - This topic discusses use limitations of vSphere vMotion, including the use of it with DRS. - -- **[Placement and Organization of Virtual Machines](gemfire_performance_on_vsphere_guidelines.html#topic_E53BBF3D09A54953B02DCE2BD00D51E0)** - - This section provides guidelines on JVM instances and placement of redundant copies of cached data. - -- **[Virtual Machine Memory Reservation](gemfire_performance_on_vsphere_guidelines.html#topic_567308E9DE07406BB5BF420BE77B6558)** - - This section provides guidelines for sizing and setting memory. - -- **[vSphere High Availability and Apache Geode](gemfire_performance_on_vsphere_guidelines.html#topic_424B940584044CF6A685E86802548A27)** - - On Apache Geode virtual machines, disable vSphere High Availability (HA). - -- **[Storage Guidelines](gemfire_performance_on_vsphere_guidelines.html#topic_913B15841C4249A68697F3D91281A645)** - - This section provides storage guidelines for persistence files, binaries, logs, and more. - -- **[Additional Resources](gemfire_performance_on_vsphere_guidelines.html#topic_628F038FD4954E56BF4192F17FD3D119)** - - VMware provides additional resources for optimizing vSphere, Java applications, and Apache Geode. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere_guidelines.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere_guidelines.html.md.erb b/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere_guidelines.html.md.erb deleted file mode 100644 index b5cb8a2..0000000 --- a/geode-docs/managing/monitor_tune/gemfire_performance_on_vsphere_guidelines.html.md.erb +++ /dev/null @@ -1,136 +0,0 @@ ---- -title: Operating System Guidelines ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> -<a id="topic_F48990A6A37144988D49E132E17E117C"></a> - - -Use the latest supported version of the guest OS, and use Java large paging. - -- **Use the latest supported version of the guest operating system**. This guideline is probably the most important. Upgrade the guest OS to a recent version supported by Geode. For example, for RHEL, use at least version 7.0 or for SLES, use at least 11.0. For Windows, use Windows Server 2012. For RedHat Linux users, it is particularly beneficial to use RHEL 7 since there are specific enhancements in the RHEL 7 release that improve virtualized latency sensitive workloads. -- **Use Java large paging in guest OS**. Configure Java on the guest OS to use large pages. Add the following command line option when launching Java: - - ``` pre - -XX:+UseLargePages - ``` - -## <a id="topic_D8393B1A75364E46B0F959F0DE820E9E" class="no-quick-link"></a>NUMA, CPU, and BIOS Settings - -This section provides VMware- recommended NUMA, CPU, and BIOS settings for your hardware and virtual machines. - -- Always enable hyper-threading, and do not overcommit CPU. -- For most production Apache Geode servers, always use virtual machines with at least two vCPUs . -- Apply non-uniform memory access (NUMA) locality by sizing virtual machines to fit within the NUMA node. -- VMware recommends the following BIOS settings: - - **BIOS Power Management Mode:** Maximum Performance. - - **CPU Power and Performance Management Mode:** Maximum Performance. - - **Processor Settings:**Turbo Mode enabled. - - **Processor Settings:**C States disabled. - -**Note:** -Settings may vary slightly depending on your hardware make and model. Use the settings above or equivalents as needed. - -## <a id="topic_7A5F1EAD7A6C4E21BB1FF7CF3B625BC5" class="no-quick-link"></a>Physical and Virtual NIC Settings - -These guidelines help you reduce latency. - -- **Physical NIC:** VMware recommends that you disable interrupt coalescing on the physical NIC of your ESXi host by using the following command: - - ``` pre - ethtool -C vmnicX rx-usecs 0 rx-frames 1 rx-usecs-irq 0 rx-frames-irq 0 - ``` - - where `vmnicX` is the physical NIC as reported by the ESXi command: - - ``` pre - esxcli network nic list - ``` - - You can verify that your settings have taken effect by issuing the command: - - ``` pre - ethtool -C vmnicX - ``` - - If you restart the ESXi host, the above configuration must be reapplied. - - **Note:** - Disabling interrupt coalescing can reduce latency in virtual machines; however, it can impact performance and cause higher CPU utilization. It can also defeat the benefits of Large Receive Offloads (LRO) because some physical NICs (such as Intel 10GbE NICs) automatically disable LRO when interrupt coalescing is disabled. See [http://kb.vmware.com/kb/1027511](http://kb.vmware.com/kb/1027511) for more details. - -- **Virtual NIC:** Use the following guidelines when configuring your virtual NICs: - - Use VMXNET3 virtual NICs for your latency-sensitive or otherwise performance-critical virtual machines. See [http://kb.vmware.com/kb/1001805](http://kb.vmware.com/kb/1001805) for details on selecting the appropriate type of virtual NIC for your virtual machine. - - VMXNET3 supports adaptive interrupt coalescing that can help drive high throughput to virtual machines that have multiple vCPUs with parallelized workloads (multiple threads), while minimizing latency of virtual interrupt delivery. However, if your workload is extremely sensitive to latency, VMware recommends that you disable virtual interrupt coalescing for your virtual NICs. You can do this programmatically via API or by editing your virtual machine's .vmx configuration file. Refer to your vSphere API Reference or VMware ESXi documentation for specific instructions. - -## <a id="topic_E6EB8AB6CCEF435A98B48B867FE9BFEB" class="no-quick-link"></a>VMware vSphere vMotion and DRS Cluster Usage - -This topic discusses use limitations of vSphere vMotion, including the use of it with DRS. - -- When you first commission the data management system, place VMware vSphere Distributed Resource Scheduler⢠(DRS) in manual mode to prevent an automatic VMware vSphere vMotion® operation that can affect response times. -- Reduce or eliminate the use of vMotion to migrate Geode virtual machines when they are under heavy load. -- Do not allow vMotion migrations with Apache Geode locator processes, as the latency introduced to this process can cause other members of the Apache Geode servers to falsely suspect that other members are dead. -- Use dedicated Apache Geode vSphere DRS clusters. This is especially important when you consider that the physical NIC and virtual NIC are specifically tuned to disable Interrupt Coalescing on every NIC of an ESXi host in the cluster. This type of tuning benefits Geode workloads, but it can hurt other non-Apache Geode workloads that are memory throughput-bound as opposed to latency sensitive as in the case of Apache Geode workloads. -- If using a dedicated vSphere DRS cluster is not an option, and Apache Geode must run in a shared DRS cluster, make sure that DRS rules are set up not to perform vMotion migrations on Geode virtual machines. -- If you must use vMotion for migration, VMware recommends that all vMotion migration activity of Apache Geode members occurs over 10GbE, during periods of low activity and scheduled maintenance windows. - -## <a id="topic_E53BBF3D09A54953B02DCE2BD00D51E0" class="no-quick-link"></a>Placement and Organization of Virtual Machines - -This section provides guidelines on JVM instances and placement of redundant copies of cached data. - -- Have one JVM instance per virtual machine. -- Increasing the heap space to service the demand for more data is better than installing a second instance of a JVM on a single virtual machine. If increasing the JVM heap size is not an option, consider placing the second JVM on a separate newly created virtual machine, thus promoting more effective horizontal scalability. As you increase the number of Apache Geode servers, also increase the number of virtual machines to maintain a 1:1:1 ratio among the Apache Geode server, the JVM, and the virtual machines. -- Size for a minimum of four vCPU virtual machines with one Apache Geode server running in one JVM instance. This allows ample CPU cycles for the garbage collector, and the rest for user transactions. -- Because Apache Geode can place redundant copies of cached data on any virtual machine, it is possible to inadvertently place two redundant data copies on the same ESX/ESXi host. This is not optimal if a host fails. To create a more robust configuration, use VM1-to-VM2 anti-affinity rules, to indicate to vSphere that VM1 and VM2 can never be placed on the same host because they hold redundant data copies. - -## <a id="topic_567308E9DE07406BB5BF420BE77B6558" class="no-quick-link"></a>Virtual Machine Memory Reservation - -This section provides guidelines for sizing and setting memory. - -- Set memory reservation at the virtual machine level so that ESXi provides and locks down the needed physical memory upon virtual machine startup. Once allocated, ESXi does not allow the memory to be taken away. -- Do not overcommit memory for Geode hosts. -- When sizing memory for a Geode server within one JVM on one virtual machine, the total reserved memory for the virtual machine should not exceed what is available within one NUMA node for optimal performance. - -## <a id="topic_424B940584044CF6A685E86802548A27" class="no-quick-link"></a>vSphere High Availability and Apache Geode - -On Apache Geode virtual machines, disable vSphere High Availability (HA). - -If you are using a dedicated Apache Geode DRS cluster, then you can disable HA across the cluster. However, if you are using a shared cluster, exclude Geode virtual machines from vSphere HA. - -Additionally, to support high availability, you can also set up anti-affinity rules between the Apache Geode virtual machines to prevent two Apache Geode servers from running on the same ESXi host within the same DRS cluster. - -## <a id="topic_913B15841C4249A68697F3D91281A645" class="no-quick-link"></a>Storage Guidelines - -This section provides storage guidelines for persistence files, binaries, logs, and more. - -- Use the PVSCSI driver for I/O intensive Apache Geode workloads. -- Align disk partitions at the VMFS and guest operating system levels. -- Provision VMDK files as eagerzeroedthick to avoid lazy zeroing for Apache Geode members. -- Use separate VMDKs for Apache Geode persistence files, binaries, and logs. -- Map a dedicated LUN to each VMDK. -- For Linux virtual machines, use NOOP scheduling as the I/O scheduler instead of Completely Fair Queuing (CFQ). Starting with the Linux kernel 2.6, CFQ is the default I/O scheduler in many Linux distributions. See [http://kb.vmware.com/kb/2011861](http://kb.vmware.com/kb/2011861) for more information. - -## <a id="topic_628F038FD4954E56BF4192F17FD3D119" class="no-quick-link"></a>Additional Resources - -VMware provides additional resources for optimizing vSphere, Java applications, and Apache Geode. - -- "Performance Best Practices for VMware vSphere 5.0" - [http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf](http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf) -- "Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere Virtual Machines" - [http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf](http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf) -- "Enterprise Java Applications on VMware - Best Practices Guide" - [http://www.vmware.com/resources/techresources/1087](http://www.vmware.com/resources/techresources/1087) -- "High Performance Data with VMware Pivotal⢠GemFire® Best Practices Guide" - [https://www.vmware.com/files/pdf/techpaper/vmw-vfabric-gemFire-best-practices-guide.pdf](https://www.vmware.com/files/pdf/techpaper/vmw-vfabric-gemFire-best-practices-guide.pdf) - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/multicast_communication.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication.html.md.erb deleted file mode 100644 index ba823c6..0000000 --- a/geode-docs/managing/monitor_tune/multicast_communication.html.md.erb +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: Multicast Communication ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -You can make configuration adjustments to improve the UDP multicast performance of peer-to-peer communication in your Geode system. - -Before you begin, you should understand Geode [Basic Configuration and Programming](../../basic_config/book_intro.html). See also the general communication tuning and UDP tuning covered in [Socket Communication](socket_communication.html) and [UDP Communication](udp_communication.html#udp_comm). - -- **[Provisioning Bandwidth for Multicast](../../managing/monitor_tune/multicast_communication_provisioning_bandwidth.html)** - - Multicast installations require more planning and configuration than TCP installations. With IP multicast, you gain scalability but lose the administrative convenience of TCP. - -- **[Testing Multicast Speed Limits](../../managing/monitor_tune/multicast_communication_testing_multicast_speed_limits.html)** - - TCP automatically adjusts its speed to the capability of the processes using it and enforces bandwidth sharing so that every process gets a turn. With multicast, you must determine and explicitly set those limits. - -- **[Configuring Multicast Speed Limits](../../managing/monitor_tune/multicast_communication_configuring_speed_limits.html)** - - After you determine the maximum transmission rate, configure and tune your production system. - -- **[Run-time Considerations for Multicast](../../managing/monitor_tune/multicast_communication_runtime_considerations.html)** - - When you use multicast for messaging and data distribution, you need to understand how the health monitoring setting works and how to control memory use. - -- **[Troubleshooting the Multicast Tuning Process](../../managing/monitor_tune/multicast_communication_troubleshooting.html)** - - Several problems may arise during the initial testing and tuning process for multicasting. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/multicast_communication_configuring_speed_limits.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication_configuring_speed_limits.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication_configuring_speed_limits.html.md.erb deleted file mode 100644 index a6cb090..0000000 --- a/geode-docs/managing/monitor_tune/multicast_communication_configuring_speed_limits.html.md.erb +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: Configuring Multicast Speed Limits ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -After you determine the maximum transmission rate, configure and tune your production system. - -<a id="multicast__section_8E225FC6829946C287552BC7996F2765"></a> -For best performance, the producer and the consumers should run on different machines and each process should have at least one CPU dedicated to it. The following is a list of configuration changes that can improve multicast performance. Check with your system administrator about changing any of the limits discussed here. - -- Increase the default datagram size for systems running Microsoft Windows from 1024 bytes to a value that matches your networkâs maximum transmission unit (MTU), which is typically 1500 bytes. The higher setting should improve the systemâs network performance. -- Distribution statistics for stack time probes are disabled by default to increase multicast performance. To reduce multicast speed, you can enable time statistics by setting the gemfire.`enable-time-statistics` property to true. - - This enables time statistics for a Java application: - - ``` pre - -Dgemfire.enable-time-statistics=true - ``` - - The time statistics properties are passed to the cache server on the `gfsh` the command line: - - ``` pre - gfsh>start server --name=server_name --enable-time-statistics=true - ``` - -- Monitor the members that receive data for signs of data loss. A few data loss messages can happen normally during region creation. Multicast retransmit requests and unicast retransmits can also be monitored to detect data loss. Even when you see data loss, the cause of the problem may have nothing to do with the network. However, if it happens constantly then you should try testing the flow control rate again -- If necessary, reconfigure all the `gemfire.properties` files and repeat with lower flow control maximum credits until you find the maximum useful rate for your installation. -- Slow system performance might be helped by reducing how far your multicast messaging goes in your network. -- Reduce multicast latency by disabling batching. By default, Geode uses batching for operations when the regionâs scope is distributed-no-ack. Set the `disableBatching` property to true on the application or when starting a cache server process through the `gfsh` command line: - - ``` pre - gfsh>start server --name=server_name --J=-Dp2p.disableBatching=true - ``` - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/multicast_communication_provisioning_bandwidth.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication_provisioning_bandwidth.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication_provisioning_bandwidth.html.md.erb deleted file mode 100644 index 402590f..0000000 --- a/geode-docs/managing/monitor_tune/multicast_communication_provisioning_bandwidth.html.md.erb +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: Provisioning Bandwidth for Multicast ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -Multicast installations require more planning and configuration than TCP installations. With IP multicast, you gain scalability but lose the administrative convenience of TCP. - -<a id="multicast__section_B7DA88707CBF4713A1E287CAA9A80EB9"></a> -When you install an application that runs over TCP, the network is almost always set up for TCP and other applications are already using it. When you install an application to run over IP multicast it may be the first multicast application on the network. - -Multicast is very dependent on the environment in which it runs. Its operation is affected by the network hardware, the network software, the machines, which Geode processes run on which machines, and whether there are any competing applications. You could find that your site has connectivity in TCP but not in multicast because some switches and network cards do not support multicast. Your network could have latent problems that you would never see otherwise. To successfully implement a distributed Geode system using multicast requires the cooperation of both system and network administrators. - -**Bounded Operation Over Multicast** - -Group rate control is required for Geode systems to maintain cache coherence. If your application delivers the same data to a group of members, your system tuning effort needs to focus on the slow receivers. - -If some of your members have trouble keeping up with the incoming data, the other members in the group may be impacted. At best, slow receivers cause the producer to use buffering, adding latency for the slow receiver and perhaps for all of them. In the worst case, throughput for the group can stop entirely while the producerâs CPU, memory and network bandwidth are dedicated to serving the slow receivers. - -To address this issue, you can implement a bounded operation policy, which sets boundaries for the producerâs operation. The appropriate rate limits are determined through tuning and testing to allow the fastest operation possible while minimizing data loss and latency in the group of consumers. This policy is suited to applications such as financial market data, where high throughput, reliable delivery and network stability are required. With the boundaries set correctly, your producerâs traffic cannot cause a network outage. - -Multicast protocols typically have a flow control protocol built into them to keep processes from being overrun. The Geode flow control protocol uses the mcast-flow-control property to set producer and consumer boundaries for multicast flow operations. The property provides these three configuration settings: - -<table> -<colgroup> -<col width="50%" /> -<col width="50%" /> -</colgroup> -<tbody> -<tr class="odd"> -<td><p><code class="ph codeph">byteAllowance</code></p></td> -<td><p>Number of bytes that can be sent without a recharge.</p></td> -</tr> -<tr class="even"> -<td><p><code class="ph codeph">rechargeThreshold</code></p></td> -<td><p>Tells consumers how low the producerâs initial to remaining allowance ratio should be before sending a recharge.</p></td> -</tr> -<tr class="odd"> -<td><code class="ph codeph">rechargeBlockMs</code></td> -<td><p>Tells the producer how long to wait for a recharge before requesting one.</p></td> -</tr> -</tbody> -</table> - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/multicast_communication_runtime_considerations.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication_runtime_considerations.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication_runtime_considerations.html.md.erb deleted file mode 100644 index 77fd42b..0000000 --- a/geode-docs/managing/monitor_tune/multicast_communication_runtime_considerations.html.md.erb +++ /dev/null @@ -1,47 +0,0 @@ ---- -title: Run-time Considerations for Multicast ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -When you use multicast for messaging and data distribution, you need to understand how the health monitoring setting works and how to control memory use. - -**Multicast Health Monitor** - -The Geode management and monitoring system is supplemented by a maxRetransmissionRatio health monitoring setting for distributed system members. This ratio is the number of retransmission requests received divided by the number of multicast datagrams written. If the ratio is at 1.0, the member is retransmitting as many packets as it originally sent. Retransmissions are point-to-point, and many processes may request retransmission, so this number can get quite high if problems occur. The default value for maxRetransmissionRatio is 0.2. - -For example, consider a distributed system with one producer and two consumers of cache events using multicast to transmit cache updates. The new member is added, which is running on a machine without multicast enabled. As a result, there is a retransmission request for every cache update, and the maxRetransmissionRatio changes to 1.0. - -**Controlling Memory Use on Geode Hosts with Multicast** - -Running out of memory can impede a memberâs performance and eventually lead to severe errors. - -When data is distributed over multicast, Geode incurs a fixed overhead of memory reserved for transmission buffers. A specified amount of memory is reserved for each distributed region. These producer-side buffers are used only when a receiver is not getting enough CPU to read from its own receiving buffer as quickly as the producer is sending. In this case, the receiver complains of lost data. The producer then retrieves the data, if it still exists in its buffer, and resends to the receiver. - -Tuning the transmission buffers requires a careful balance. Larger buffers mean that more data remains available for retransmission, providing more protection in case of a problem. On the other hand, a larger amount of reserved memory means that less memory is available for caching. - -You can adjust the transmission buffer size by resetting the mcast-send-buffer-size parameter in the `gemfire.properties` file: - -``` pre -mcast-send-buffer-size=45000 -``` - -**Note:** -The maximum buffer size is constrained only by the limits of your system. If you are not seeing problems that could be related to lack of memory then do not change the default, since it provides greater protection in case of network problems. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/multicast_communication_testing_multicast_speed_limits.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication_testing_multicast_speed_limits.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication_testing_multicast_speed_limits.html.md.erb deleted file mode 100644 index 1e8faa7..0000000 --- a/geode-docs/managing/monitor_tune/multicast_communication_testing_multicast_speed_limits.html.md.erb +++ /dev/null @@ -1,145 +0,0 @@ ---- -title: Testing Multicast Speed Limits ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -TCP automatically adjusts its speed to the capability of the processes using it and enforces bandwidth sharing so that every process gets a turn. With multicast, you must determine and explicitly set those limits. - -<a id="multicast__section_AB06591284DB4E9785EE79FBE1C59554"></a> -Without the proper configuration, multicast delivers its traffic as fast as possible, overrunning the ability of consumers to process the data and locking out other processes that are waiting for the bandwidth. You can tune your multicast and unicast behavior using mcast-flow-control in `gemfire.properties`. - -**Using Iperf** - -Iperf is an open-source TCP/UDP performance tool that you can use to find your siteâs maximum rate for data distribution over multicast. Iperf can be downloaded from web sites such as the National Laboratory for Applied Network Research (NLANR). - -Iperf measures maximum bandwidth, allowing you to tune parameters and UDP characteristics. Iperf reports statistics on bandwidth, delay jitter, and datagram loss. On Linux, you can redirect this output to a file; on Windows, use the -o filename parameter. - -Run each test for ten minutes to make sure any potential problems have a chance to develop. Use the following command lines to start the sender and receivers. - -**Sender**: - -``` pre -iperf -c 192.0.2.0 -u -T 1 -t 100 -i 1 -b 1000000000 -``` - -where: - -<table> -<colgroup> -<col width="50%" /> -<col width="50%" /> -</colgroup> -<tbody> -<tr class="odd"> -<td>-c address</td> -<td><p>Run in client mode and connect to a multicast address</p></td> -</tr> -<tr class="even"> -<td>-u</td> -<td><p>Use UDP</p></td> -</tr> -<tr class="odd"> -<td>-T #</td> -<td><p>Multicast time-to-live: number of subnets across which a multicast packet can travel before the routers drop the packet</p></td> -</tr> -</tbody> -</table> - -**Note:** -Do not set the -T parameter above 1 without consulting your network administrator. If this number is too high then the iperf traffic could interfere with production applications or continue out onto the internet. - -<table> -<colgroup> -<col width="50%" /> -<col width="50%" /> -</colgroup> -<tbody> -<tr class="odd"> -<td>-t</td> -<td><p>Length of time to transmit, in seconds</p></td> -</tr> -<tr class="even"> -<td>-i</td> -<td><p>Time between periodic bandwidth reports, in seconds</p></td> -</tr> -<tr class="odd"> -<td>-b</td> -<td>Sending bandwidth, in bits per second</td> -</tr> -</tbody> -</table> - -**Receiver**: - -``` pre -iperf -s -u -B 192.0.2.0 -i 1 -``` - -where: - -<table> -<colgroup> -<col width="50%" /> -<col width="50%" /> -</colgroup> -<tbody> -<tr class="odd"> -<td><p>-s</p></td> -<td><p>Run in server mode</p></td> -</tr> -<tr class="even"> -<td><p>-u</p></td> -<td><p>Use UDP</p></td> -</tr> -<tr class="odd"> -<td><p>-B address</p></td> -<td><p>Bind to a multicast address</p></td> -</tr> -<tr class="even"> -<td>-i #</td> -<td>Time between periodic bandwidth reports, in seconds</td> -</tr> -</tbody> -</table> - -**Note:** -If your Geode distributed system runs across several subnets, start a receiver on each subnet. - -In the receiverâs output, look at the Lost/Total Datagrams columns for the number and percentage of lost packets out of the total sent. - -**Output From Iperf Testing**: - -``` pre -[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams -[ 3] 0.0- 1.0 sec 129 KBytes 1.0 Mbits/sec 0.778 ms 61/ 151 (40%) -[ 3] 1.0- 2.0 sec 128 KBytes 1.0 Mbits/sec 0.236 ms 0/ 89 (0%) -[ 3] 2.0- 3.0 sec 128 KBytes 1.0 Mbits/sec 0.264 ms 0/ 89 (0%) -[ 3] 3.0- 4.0 sec 128 KBytes 1.0 Mbits/sec 0.248 ms 0/ 89 (0%) -[ 3] 0.0- 4.3 sec 554 KBytes 1.0 Mbits/sec 0.298 ms 61/ 447 (14%) -``` - -Rerun the test at different bandwidths until you find the maximum useful multicast rate. Start high, then gradually decrease the send rate until the test runs consistently with no packet loss. For example, you might need to run five tests in a row, changing the -b (bits per second) parameter each time until there is no loss: - -1. -b 1000000000 (loss) -2. -b 900000000 (no loss) -3. -b 950000000 (no loss) -4. -b 980000000 (a bit of loss) -5. -b 960000000 (no loss) - -Enter iperf -h to see all of the command-line options. For more information, see the Iperf user manual. http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/multicast_communication_troubleshooting.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/multicast_communication_troubleshooting.html.md.erb b/geode-docs/managing/monitor_tune/multicast_communication_troubleshooting.html.md.erb deleted file mode 100644 index c7f9c48..0000000 --- a/geode-docs/managing/monitor_tune/multicast_communication_troubleshooting.html.md.erb +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Troubleshooting the Multicast Tuning Process ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -Several problems may arise during the initial testing and tuning process for multicasting. - -**Some or All Members Cannot Communicate** - -If your applications and cache servers cannot talk to each other, even though they are configured correctly, you may not have multicast connectivity on your network. Itâs common to have unicast connectivity, but not multicast connectivity. See your network administrator. - -**Multicast Is Slower Than Expected** - -Look for an Ethernet flow control limit. If you have mixed-speed networks that result in a multicast flooding problem, the Ethernet hardware may be trying to slow down the fast traffic. - -Make sure your network hardware can deal with multicast traffic and route it efficiently. Some network hardware designed to handle multicast does not perform well enough to support a full-scale production system. - -**Multicast Fails Unexpectedly** - -If you find through testing that multicast fails above a round number, for example, it works up to 100 Mbps and fails at all rates over that, suspect that it is failing because it exceeds the network rate. This problem often arises at sites where one of the secondary LANs is slower than the main network - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/performance_controls.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/performance_controls.html.md.erb b/geode-docs/managing/monitor_tune/performance_controls.html.md.erb deleted file mode 100644 index ddc713c..0000000 --- a/geode-docs/managing/monitor_tune/performance_controls.html.md.erb +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: Performance Controls ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -This topic provides tuning suggestions of particular interest to developers, primarily programming techniques and cache configuration. - -Before you begin, you should understand Apache Geode [Basic Configuration and Programming](../../basic_config/book_intro.html). - -- **[Data Serialization](../../managing/monitor_tune/performance_controls_data_serialization.html)** - - In addition to standard Java serialization, Geode offers serialization options that give you higher performance and greater flexibility for data storage, transfers, and language types. - -- **[Setting Cache Timeouts](../../managing/monitor_tune/performance_controls_setting_cache_timeouts.html)** - - Cache timeout properties can modified through the gfsh `alter runtime` command (or declared in the `cache.xml` file) and can also be set through methods of the interface, `org.apache.geode.cache.Cache`. - -- **[Controlling Socket Use](../../managing/monitor_tune/performance_controls_controlling_socket_use.html)** - - For peer-to-peer communication, you can manage socket use at the system member level and at the thread level. - -- **[Management of Slow Receivers](../../managing/monitor_tune/performance_controls_managing_slow_receivers.html)** - - You have several options for handling slow members that receive data distribution. The slow receiver options control only to peer-to-peer communication between distributed regions using TCP/IP. This topic does not apply to client/server or multi-site communication, or to communication using the UDP unicast or IP multicast protocols. - -- **[Increasing the Ratio of Cache Hits](../../managing/monitor_tune/performance_controls_increasing_cache_hits.html)** - - The more frequently a get fails to find a valid value in the first cache and has to try a second cache, the more the overall performance is affected. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/performance_controls_controlling_socket_use.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/performance_controls_controlling_socket_use.html.md.erb b/geode-docs/managing/monitor_tune/performance_controls_controlling_socket_use.html.md.erb deleted file mode 100644 index d4327eb..0000000 --- a/geode-docs/managing/monitor_tune/performance_controls_controlling_socket_use.html.md.erb +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: Controlling Socket Use ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -For peer-to-peer communication, you can manage socket use at the system member level and at the thread level. - -The conserve-sockets setting indicates whether application threads share sockets with other threads or use their own sockets for distributed system member communication. This setting has no effect on communication between a server and its clients, but it does control the serverâs communication with its peers or a gateway sender's communication with a gateway receiver. In client/server settings in particular, where there can be a large number of clients for each server, controlling peer-to-peer socket use is an important part of tuning server performance. - -You configure conserve-sockets for the member as a whole in `gemfire.properties`. Additionally, you can change the sockets conservation policy for the individual thread through the API. - -When conserve-sockets is set to false, each application thread uses a dedicated thread to send to each of its peers and a dedicated thread to receive from each peer. Disabling socket conservation requires more system resources, but can potentially improve performance by removing socket contention between threads and optimizing distributed ACK operations. For distributed regions, the put operation, and destroy and invalidate for regions and entries, can all be optimized with conserve-sockets set to false. For partitioned regions, setting conserve-sockets to false can improve general throughput. - -**Note:** -When you have transactions operating on EMPTY, NORMAL or PARTITION regions, make sure that `conserve-sockets` is set to false to avoid distributed deadlocks. - -You can override the `conserve-sockets` setting for individual threads. These methods are in `org.apache.geode.distributed.DistributedSystem`: - -- `setThreadsSocketPolicy`. Sets the calling threadâs individual socket policy, overriding the policy set for the application as a whole. If set to true, the calling thread shares socket connections with other threads. If false, the calling thread has its own sockets. -- `releaseThreadsSockets`. Frees any sockets held by the calling thread. Threads hold their own sockets only when conserve-sockets is false. Threads holding their own sockets can call this method to avoid holding the sockets until the socket-lease-time has expired. - -A typical implementation might set conserve-sockets to true at the application level and then override the setting for the specific application threads that perform the bulk of the distributed operations. The example below shows an implementation of the two API calls in a thread that performs benchmark tests. The example assumes the class implements Runnable. Note that the invocation, setThreadsSocketPolicy(false), is only meaningful if conserve-sockets is set to true at the application level. - -``` pre -public void run() { - DistributedSystem.setThreadsSocketPolicy(false); - try { - // do your benchmark work - } finally { - DistributedSystem.releaseThreadsSockets(); - } -} -``` - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/performance_controls_data_serialization.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/performance_controls_data_serialization.html.md.erb b/geode-docs/managing/monitor_tune/performance_controls_data_serialization.html.md.erb deleted file mode 100644 index 139f1bb..0000000 --- a/geode-docs/managing/monitor_tune/performance_controls_data_serialization.html.md.erb +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: Data Serialization ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -In addition to standard Java serialization, Geode offers serialization options that give you higher performance and greater flexibility for data storage, transfers, and language types. - -Under *Developing with Apache Geode*, see [Data Serialization](../../developing/data_serialization/chapter_overview.html#data_serialization). - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/performance_controls_increasing_cache_hits.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/performance_controls_increasing_cache_hits.html.md.erb b/geode-docs/managing/monitor_tune/performance_controls_increasing_cache_hits.html.md.erb deleted file mode 100644 index c3a4d13..0000000 --- a/geode-docs/managing/monitor_tune/performance_controls_increasing_cache_hits.html.md.erb +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: Increasing the Ratio of Cache Hits ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -The more frequently a get fails to find a valid value in the first cache and has to try a second cache, the more the overall performance is affected. - -A common cause of misses is expiration or eviction of the entry. If you have a regionâs entry expiration or eviction enabled, monitor the region and entry statistics. - -If you see a high ratio of misses to hits on the entries, consider increasing the expiration times or the maximum values for eviction, if possible. See [Eviction](../../developing/eviction/chapter_overview.html) for more information. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/performance_controls_managing_slow_receivers.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/performance_controls_managing_slow_receivers.html.md.erb b/geode-docs/managing/monitor_tune/performance_controls_managing_slow_receivers.html.md.erb deleted file mode 100644 index 30293d0..0000000 --- a/geode-docs/managing/monitor_tune/performance_controls_managing_slow_receivers.html.md.erb +++ /dev/null @@ -1,73 +0,0 @@ ---- -title: Management of Slow Receivers ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -You have several options for handling slow members that receive data distribution. The slow receiver options control only to peer-to-peer communication between distributed regions using TCP/IP. This topic does not apply to client/server or multi-site communication, or to communication using the UDP unicast or IP multicast protocols. - -Most of the options for handling slow members are related to on-site configuration during system integration and tuning. For this information, see [Slow Receivers with TCP/IP](slow_receivers.html). - -Slowing is more likely to occur when applications run many threads, send large messages (due to large entry values), or have a mix of region configurations. - -**Note:** -If you are experiencing slow performance and are sending large objects (multiple megabytes), before implementing these slow receiver options make sure your socket buffer sizes are large enough for the objects you distribute. The socket buffer size is set using gemfire.socket-buffer-size. - -By default, distribution between system members is performed synchronously. With synchronous communication, when one member is slow to receive, it can cause its producer members to slow down as well. This can lead to general performance problems in the distributed system. - -The specifications for handling slow receipt primarily affect how your members manage distribution for regions with distributed-no-ack scope, but it can affect other distributed scopes as well. If no regions have distributed-no-ack scope, this mechanism is unlikely to kick in at all. When slow receipt handling does kick in, however, it affects all distribution between the producer and consumer, regardless of scope. Partitioned regions ignore the scope attribute, but for the purposes of this discussion you should think of them as having an implicit distributed-ack scope. - -**Configuration Options** - -The slow receiver options are set in the producer memberâs region attribute, enable-async-conflation, and in the consumer memberâs async\* `gemfire.properties` settings. - -**Delivery Retries** - -If the receiver fails to receive a message, the sender continues to attempt to deliver the message as long as the receiving member is still in the distributed system. During the retry cycle, throws warnings that include this string: - -``` pre -will reattempt -``` - -The warnings are followed by an info message when the delivery finally succeeds. - -**Asynchronous Queueing For Slow Receivers** - -Your consumer members can be configured so that their producers switch to asynchronous messaging if the consumers are slow to respond to cache message distribution. - -When a producer switches, it creates a queue to hold and manage that consumerâs cache messages. When the queue empties, the producer switches back to synchronous messaging for the consumer. The settings that cause the producers to switch are specified on the consumer side in `gemfire.properties` file settings. - -If you configure your consumers for slow receipt queuing, and your region scope is distributed-no-ack, you can also configure the producer to conflate entry update messages in its queues. This configuration option is set as the region attribute enable-async-conflation. By default distributed-no-ack entry update messages are not conflated. - -Depending on the application, conflation can greatly reduce the number of messages the producer needs to send to the consumer. With conflation, when an entry update is added to the queue, if the last operation queued for that key is also an update operation, the previously enqueued update is removed, leaving only the latest update to be sent to the consumer. Only entry update messages originating in a region with distributed-no-ack scope are conflated. Region operations and entry operations other than updates are not conflated. - -<img src="../../images_svg/async_system_queue_conflation.svg" id="perf__image_0FD90F27762F4440B9ECC40803988038" class="image" /> - -Some conflation may not occur because entry updates are sent to the consumer before they can be conflated. For this example, assume no messages are sent while the update for Key A is added. - -**Note:** -This method of conflation behaves the same as server-to-client conflation. - -You can enable queue conflation on a region-by-region basis. You should always enable it unless it is incompatible with your application needs. Conflation reduces the amount of data queued and distributed. - -These are reasons why conflation might not work for your application: - -- With conflation, earlier entry updates are removed from the queue and replaced by updates sent later in the queue. This is problematic for applications that depend on a specific ordering of entry modifications. For example, if your receiver has a CacheListener that needs to know about every state change, you should disable conflation. -- If your queue remains in use for a significant period and you have entries that are updated frequently, you could have a series of update message replacements resulting in a notable delay in the arrival of any update for some entries. Imagine that update 1, before it is sent, is removed in favor of a later update 2. Then, before update 2 can be sent, it is removed in favor of update 3, and so on. This could result in unacceptably stale data on the receiver. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/performance_controls_setting_cache_timeouts.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/performance_controls_setting_cache_timeouts.html.md.erb b/geode-docs/managing/monitor_tune/performance_controls_setting_cache_timeouts.html.md.erb deleted file mode 100644 index ca69a49..0000000 --- a/geode-docs/managing/monitor_tune/performance_controls_setting_cache_timeouts.html.md.erb +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Setting Cache Timeouts ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -Cache timeout properties can modified through the gfsh `alter runtime` command (or declared in the `cache.xml` file) and can also be set through methods of the interface, `org.apache.geode.cache.Cache`. - -To modify cache timeout properties, you can issue the following `gfsh alter runtime` command. For example: - -``` pre -gfsh>alter runtime --search-timeout=150 -``` - -The `--search-timeout` parameter specifies how long a netSearch operation can wait for data before timing out. The default is 5 minutes. You may want to change this based on your knowledge of the network load or other factors. - -The next two configurations describe timeout settings for locking in regions with global scope. Locking operations can time out in two places: when waiting to obtain a lock (lock time out); and when holding a lock (lock lease time). Operations that modify objects in a global region use automatic locking. In addition, you can manually lock a global region and its entries through `org.apache.geode.cache.Region`. The explicit lock methods provided by the APIs allow you to specify a lock timeout parameter. The lock time out for implicit operations and the lock lease time for implicit and explicit operations are governed by these cache-wide settings: - -``` pre -gfsh>alter runtime --lock-timeout=30 --lock-lease=60 -``` - -- `--lock-timeout`. Timeout for object lock requests, specified in seconds. The setting affects automatic locking only, and does not apply to manual locking. The default is 1 minute. If a lock request does not return before the specified timeout period, it is cancelled and returns with a failure. -- `--lock-lease`. Timeout for object lock leases, specified in seconds. The setting affects both automatic locking and manual locking. The default is 2 minutes. Once a lock is obtained, it may remain in force for the lock lease time period before being automatically cleared by the system. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/slow_messages.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/slow_messages.html.md.erb b/geode-docs/managing/monitor_tune/slow_messages.html.md.erb deleted file mode 100644 index 7150802..0000000 --- a/geode-docs/managing/monitor_tune/slow_messages.html.md.erb +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Slow distributed-ack Messages ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -In systems with distributed-ack regions, a sudden large number of distributed-no-ack operations can cause distributed-ack operations to take a long time to complete. - -The `distributed-no-ack` operations can come from anywhere. They may be updates to `distributed-no-ack` regions or they may be other `distributed-no-ack` operations, like destroys, performed on any region in the cache, including the `distributed-ack` regions. - -The main reasons why a large number of `distributed-no-ack` messages may delay `distributed-ack` operations are: - -- For any single socket connection, all operations are executed serially. If there are any other operations buffered for transmission when a `distributed-ack` is sent, the `distributed-ack` operation must wait to get to the front of the line before being transmitted. Of course, the operationâs calling process is also left waiting. -- The `distributed-no-ack` messages are buffered by their threads before transmission. If many messages are buffered and then sent to the socket at once, the line for transmission might be very long. - -You can take these steps to reduce the impact of this problem: - -1. If youâre using TCP, check whether you have socket conservation enabled for your members. It is configured by setting the Geode property `conserve-sockets` to true. If enabled, each applicationâs threads will share sockets unless you override the setting at the thread level. Work with your application programmers to see whether you might disable sharing entirely or at least for the threads that perform `distributed-ack` operations. These include operations on `distributed-ack` regions and also `netSearches` performed on regions of any distributed scope. (Note: `netSearch` is only performed on regions with a data-policy of empty, normal and preloaded.) If you give each thread that performs `distributed-ack` operations its own socket, you effectively let it scoot to the front of the line ahead of the `distributed-no-ack` operations that are being performed by other threads. The thread-level override is done by calling the `DistributedSystem.setThreadsSocketPolicy(false)` metho d. -2. Reduce your buffer sizes to slow down the distributed-no-ack operations. These changes slow down the threads performing distributed-no-ack operations and allow the thread doing the distributed-ack operations to be sent in a more timely manner. - - If you're using UDP (you either have multicast enabled regions or have set `disable-tcp` to true in gemfire.properties), consider reducing the byteAllowance of mcast-flow-control to something smaller than the default of 3.5 megabytes. - - If you're using TCP/IP, reduce the `socket-buffer-size` in gemfire.properties. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/slow_receivers.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/slow_receivers.html.md.erb b/geode-docs/managing/monitor_tune/slow_receivers.html.md.erb deleted file mode 100644 index 5e4aebe..0000000 --- a/geode-docs/managing/monitor_tune/slow_receivers.html.md.erb +++ /dev/null @@ -1,34 +0,0 @@ ---- -title: Slow Receivers with TCP/IP ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -You have several options for preventing situations that can cause slow receivers of data distributions. The slow receiver options control only peer-to-peer communication using TCP/IP. This discussion does not apply to client/server or multi-site communication, or to communication using the UDP unicast or multicast protocols. - -Before you begin, you should understand Geode [Basic Configuration and Programming](../../basic_config/book_intro.html). - -- **[Preventing Slow Receivers](../../managing/monitor_tune/slow_receivers_preventing_problems.html)** - - During system integration, you can identify and eliminate potential causes of slow receivers in peer-to-peer communication. - -- **[Managing Slow Receivers](../../managing/monitor_tune/slow_receivers_managing.html)** - - If the receiver fails to receive a message, the sender continues to attempt to deliver the message as long as the receiving member is still in the distributed system. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/84cfbdfc/geode-docs/managing/monitor_tune/slow_receivers_managing.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/managing/monitor_tune/slow_receivers_managing.html.md.erb b/geode-docs/managing/monitor_tune/slow_receivers_managing.html.md.erb deleted file mode 100644 index 49e93c4..0000000 --- a/geode-docs/managing/monitor_tune/slow_receivers_managing.html.md.erb +++ /dev/null @@ -1,116 +0,0 @@ ---- -title: Managing Slow Receivers ---- - -<!-- -Licensed to the Apache Software Foundation (ASF) under one or more -contributor license agreements. See the NOTICE file distributed with -this work for additional information regarding copyright ownership. -The ASF licenses this file to You under the Apache License, Version 2.0 -(the "License"); you may not use this file except in compliance with -the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> - -If the receiver fails to receive a message, the sender continues to attempt to deliver the message as long as the receiving member is still in the distributed system. - -During the retry cycle, Geode throws warnings that include this string: - -``` pre -will reattempt -``` - -The warnings are followed by an informational message when the delivery finally succeeds. - -For distributed regions, the scope of a region determines whether distribution acknowledgments and distributed synchronization are required. Partitioned regions ignore the scope attribute, but for the purposes of this discussion you should think of them as having an implicit distributed-ack scope. - -By default, distribution between system members is performed synchronously. With synchronous communication, when one member is slow to receive, it can cause its producers to slow down as well. This, of course, can lead to general performance problems in the distributed system. - -If you are experiencing slow performance and are sending large objects (multiple megabytes), before implementing these slow receiver options make sure your socket buffer sizes are appropriate for the size of the objects you distribute. The socket buffer size is set using socket-buffer-size in the `gemfire.properties` file. - -**Managing Slow distributed-no-ack Receivers** - -You can configure your consumer members so their messages are queued separately when they are slow to respond. The queueing happens in the producer members when the producers detect slow receipt and allows the producers to keep sending to other consumers at a normal rate. Any member that receives data distribution can be configured as described in this section. - -The specifications for handling slow receipt primarily affect how your members manage distribution for regions with distributed-no-ack scope, where distribution is asynchronous, but the specifications can affect other distributed scopes as well. If no regions have distributed-no-ack scope, the mechanism is unlikely to kick in at all. When slow receipt handling does kick in, however, it affects all distribution between the producer and that consumer, regardless of scope. - -**Note:** -These slow receiver options are disabled in systems using SSL. See [SSL](../security/ssl_overview.html). - -Each consumer member determines how its own slow behavior is to be handled by its producers. The settings are specified as distributed system connection properties. This section describes the settings and lists the associated properties. - -- async-distribution-timeoutâThe distribution timeout specifies how long producers are to wait for the consumer to respond to synchronous messaging before switching to asynchronous messaging with that consumer. When a producer switches to asynchronous messaging, it creates a queue for that consumerâs messages and a separate thread to handle the communication. When the queue empties, the producer automatically switches back to synchronous communication with the consumer. These settings affect how long your producerâs cache operations might block. The sum of the timeouts for all consumers is the longest time your producer might block on a cache operation. -- async-queue-timeoutâThe queue timeout sets a limit on the length of time the asynchronous messaging queue can exist without a successful distribution to the slow receiver. When the timeout is reached, the producer asks the consumer to leave the distributed system. -- async-max-queue-sizeâThe maximum queue size limits the amount of memory the asynchronous messaging queue can consume. When the maximum is reached, the producer asks the consumer to leave the distributed system. - -**Configuring Async Queue Conflation** - -When the scope is distributed-no-ack scope, you can configure the producer to conflate entry update messages in its queues, which may further speed communication. By default, distributed-no-ack entry update messages are not conflated. The configuration is set in the producer at the region level. - -**Forcing the Slow Receiver to Disconnect** - -If either of the queue timeout or maximum queue size limits is reached, the producer sends the consumer a high-priority message (on a different TCP connection than the connection used for cache messaging) telling it to disconnect from the distributed system. This prevents growing memory consumption by the other processes that are queuing changes for the slow receiver while they wait for that receiver to catch up. It also allows the slow member to start fresh, possibly clearing up the issues that were causing it to run slowly. - -When a producer gives up on a slow receiver, it logs one of these types of warnings: - -- Blocked for time ms which is longer than the max of asyncQueueTimeout ms so asking slow receiver slow\_receiver\_ID to disconnect. -- Queued bytes exceed max of asyncMaxQueueSize so asking slow receiver slow\_receiver\_ID to disconnect. - -When a process disconnects after receiving a request to do so by a producer, it logs a warning message of this type: - -- Disconnect forced by producer because we were too slow. - -These messages only appear in your logs if logging is enabled and the log level is set to a level that includes warning (which it does by default). See [Logging](../logging/logging.html#concept_30DB86B12B454E168B80BB5A71268865). - -If your consumer is unable to receive even high priority messages, only the producerâs warnings will appear in the logs. If you see only producer warnings, you can restart the consumer process. Otherwise, the Geode failure detection code will eventually cause the member to leave the distributed system on its own. - -**Use Cases** - -These are the main use cases for the slow receiver specifications: - -- Message burstsâWith message bursts, the socket buffer can overflow and cause the producer to block. To keep from blocking, first make sure your socket buffer is large enough to handle a normal number of messages (using the socket-buffer-size property), then set the async distribution timeout to 1. With this very low distribution timeout, when your socket buffer does fill up, the producer quickly switches to async queueing. Use the distribution statistics, asyncQueueTimeoutExceeded and asyncQueueSizeExceeded, to make sure your queue settings are high enough to avoid forcing unwanted disconnects during message bursts. -- Unhealthy or dead membersâWhen members are dead or very unhealthy, they may not be able to communicate with other distributed system members. The slow receiver specifications allow you to force crippled members to disconnect, freeing up resources and possibly allowing the members to restart fresh. To configure for this, set the distribution timeout high (one minute), and set the queue timeout low. This is the best way to avoid queueing for momentary slowness, while still quickly telling very unhealthy members to leave the distributed system. -- Combination message bursts and unhealthy membersâTo configure for both of the above situations, set the distribution timeout low and the queue timeout high, as for the message bursts scenario. - -**Managing Slow distributed-ack Receivers** - -When using a distribution scope other than distributed-no-ack, alerts are issued for slow receivers. A member that isnât responding to messages may be sick, slow, or missing. Sick or slow members are detected in message transmission and reply-wait processing code, triggering a warning alert first. If a member still isnât responding, a severe warning alert is issued, indicating that the member may be disconnected from the distributed system. This alert sequence is enabled by setting the ack-wait-threshold and the ack-severe-alert-threshold to some number of seconds. - -When ack-severe-alert-threshold is set, regions are configured to use ether distributed-ack or global scope, or use the partition data policy. Geode will wait for a total of ack-wait-threshold seconds for a response to a cache operation, then it logs a warning alert ("Membership: requesting removal of entry(\#). Disconnected as a slow-receiver"). After waiting an additional ack-severe-alert-threshold seconds after the first threshold is reached, the system also informs the failure detection mechanism that the receiver is suspect and may be disconnected, as shown in the following figure. - -<img src="../../images_svg/member_severe_alert.svg" id="slow_recv__image_BA474143B16744F28DE0AB1CAD00FB48" class="image" /> -The events occur in this order: - -1. CACHE\_OPERATION - transmission of cache operation is initiated. -2. SUSPECT - identified as a suspect by ack-wait-threshold, which is the maximum time to wait for an acknowledge before initiating failure detection. -3. I AM ALIVE - notification to the system in response to failure detection queries, if the process is still alive. A new membership view is sent to all members if the suspect process fails to answer with I AM ALIVE. -4. SEVERE ALERT- the result of ack-severe-wait-threshold elapsing without receiving a reply. - -When a member fails suspect processing, its cache is closed and its CacheListeners are notified with the afterRegionDestroyed notification. The RegionEvent passed with this notification has a CACHE\_CLOSED operation and a FORCED\_DISCONNECT operation, as shown in the FORCED\_DISCONNECT example. - -``` pre -public static final Operation FORCED_DISCONNECT -= new Operation("FORCED_DISCONNECT", - true, // isLocal - true, // isRegion - OP_TYPE_DESTROY, - OP_DETAILS_NONE - ); - -``` - -A cache closes due to being expelled from the distributed system by other members. Typically, this happens when a member becomes unresponsive and does not respond to heartbeat requests within the member-timeout period, or when ack-severe-alert-threshold has expired without a response from the member. - -**Note:** -This is marked as a region operation. - -Other members see the normal membership notifications for the departing member. For instance, RegionMembershipListeners receive the afterRemoteRegionCrashed notification, and SystemMembershipListeners receive the memberCrashed notification. - -
