http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/conflate_multisite_gateway_queue.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/conflate_multisite_gateway_queue.html.md.erb b/developing/events/conflate_multisite_gateway_queue.html.md.erb deleted file mode 100644 index e2e7ff1..0000000 --- a/developing/events/conflate_multisite_gateway_queue.html.md.erb +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: Conflating Events in a Queue ---- - -Conflating a queue improves distribution performance. When conflation is enabled, only the latest queued value is sent for a particular key. - -<a id="conflate_multisite_gateway_queue__section_294AD2E2328E4D6B8D6A73966F7B3B14"></a> -**Note:** -Do not use conflation if your receiving applications depend on the specific ordering of entry modifications, or if they need to be notified of every change to an entry. - -Conflation is most useful when a single entry is updated frequently, but other sites only need to know the current value of the entry (rather than the value of each update). When an update is added to a queue that has conflation enabled, if there is already an update message in the queue for the entry key, then the existing message assumes the value of the new update and the new update is dropped, as shown here for key A. - -<img src="../../images/MultiSite-4.gif" id="conflate_multisite_gateway_queue__image_27219DAAB6D643348641389DBAEA1E94" class="image" /> - -**Note:** -This method of conflation is different from the one used for server-to-client subscription queue conflation and peer-to-peer distribution within a distributed system. - -## <a id="conflate_multisite_gateway_queue__section_207FA6BF0F734F9A91EAACB136F8D6B5" class="no-quick-link"></a>ExamplesâConfiguring Conflation for a Gateway Sender Queue - -To enable conflation for a gateway sender queue, use one of the following mechanisms: - -- **cache.xml configuration** - - ``` pre - <cache> - <gateway-sender id="NY" parallel="true" - remote-distributed-system-id="1" - enable-persistence="true" - disk-store-name="gateway-disk-store" - enable-batch-conflation="true"/> - ... - </cache> - ``` - -- **Java API configuration** - - ``` pre - Cache cache = new CacheFactory().create(); - - GatewaySenderFactory gateway = cache.createGatewaySenderFactory(); - gateway.setParallel(true); - gateway.setPersistenceEnabled(true); - gateway.setDiskStoreName("gateway-disk-store"); - gateway.setBatchConflationEnabled(true); - GatewaySender sender = gateway.create("NY", "1"); - sender.start(); - ``` - - Entry updates in the current, in-process batch are not eligible for conflation. - -- **gfsh:** - - ``` pre - gfsh>create gateway-sender --id="NY" --parallel=true - --remote-distributed-system-id="1" - --enable-persistence=true - --disk-store-name="gateway-disk-store" - --enable-batch-conflation=true - ``` - -The following examples show how to configure conflation for an asynchronous event queue: - -- **cache.xml configuration** - - ``` pre - <cache> - <async-event-queue id="sampleQueue" persistent="true" - disk-store-name="async-disk-store" parallel="false" - enable-batch-conflation="true"> - <async-event-listener> - <class-name>MyAsyncEventListener</class-name> - <parameter name="url"> - <string>jdbc:db2:SAMPLE</string> - </parameter> - <parameter name="username"> - <string>gfeadmin</string> - </parameter> - <parameter name="password"> - <string>admin1</string> - </parameter> - </async-event-listener> - </async-event-queue> - ... - </cache> - ``` - -- **Java API configuration** - - ``` pre - Cache cache = new CacheFactory().create(); - AsyncEventQueueFactory factory = cache.createAsyncEventQueueFactory(); - factory.setPersistent(true); - factory.setDiskStoreName("async-disk-store"); - factory.setParallel(false); - factory.setBatchConflationEnabled(true); - AsyncEventListener listener = new MyAsyncEventListener(); - AsyncEventQueue sampleQueue = factory.create("customerWB", listener); - ``` - - Entry updates in the current, in-process batch are not eligible for conflation. - -- **gfsh:** - - ``` pre - gfsh>create async-event-queue --id="sampleQueue" --persistent=true - --disk-store="async-disk-store" --parallel="false" - --listener=myAsyncEventListener - --listener-param=url#jdbc:db2:SAMPLE - --listener-param=username#gfeadmin - --listener-param=password#admin1 - ``` - -
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/conflate_server_subscription_queue.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/conflate_server_subscription_queue.html.md.erb b/developing/events/conflate_server_subscription_queue.html.md.erb deleted file mode 100644 index f6d4990..0000000 --- a/developing/events/conflate_server_subscription_queue.html.md.erb +++ /dev/null @@ -1,36 +0,0 @@ ---- -title: Conflate the Server Subscription Queue ---- - -<a id="conflate_the_server_subscription_queue__section_1791DFB89502480EB57F81D16AC0EBAC"></a> -Conflating the server subscription queue can save space in the server and time in message processing. - -Enable conflation at the server level in the server region configuration: - -``` pre -<region ... > - <region-attributes enable-subscription-conflation="true" /> -</region> -``` - -Override the server setting as needed, on a per-client basis, in the clientâs `gemfire.properties`: - -``` pre -conflate-events=false -``` - -Valid `conflate-events` settings are: -- `server`, which uses the server settings -- `true`, which conflates everything sent to the client -- `false`, which does not conflate anything sent to this client - -Conflation can both improve performance and reduce the amount of memory required on the server for queuing. The client receives only the latest available update in the queue for a particular entry key. Conflation is disabled by default. - -Conflation is particularly useful when a single entry is updated often and the intermediate updates donât require processing by the client. With conflation, if an entry is updated and there is already an update in the queue for its key, the existing update is removed and the new update is placed at the end of the queue. Conflation is only done on messages that are not in the process of being sent to the client. - -<img src="../../images/ClientServerAdvancedTopics-7.gif" id="conflate_the_server_subscription_queue__image_FA77FD2857464D17BF2ED5B3CC62687A" class="image" /> - -**Note:** -This method of conflation is different from the one used for multi-site gateway sender queue conflation. It is the same as the method used for the conflation of peer-to-peer distribution messages within a single distributed system. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/event_handler_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/event_handler_overview.html.md.erb b/developing/events/event_handler_overview.html.md.erb deleted file mode 100644 index c020e3a..0000000 --- a/developing/events/event_handler_overview.html.md.erb +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: Implementing Geode Event Handlers ---- - -You can specify event handlers for region and region entry operations and for administrative events. - -- **[Implementing Cache Event Handlers](implementing_cache_event_handlers.html)** - - Depending on your installation and configuration, cache events can come from local operations, peers, servers, and remote sites. Event handlers register their interest in one or more events and are notified when the events occur. - -- **[Implementing an AsyncEventListener for Write-Behind Cache Event Handling](implementing_write_behind_event_handler.html)** - - An `AsyncEventListener` asynchronously processes batches of events after they have been applied to a region. You can use an `AsyncEventListener` implementation as a write-behind cache event handler to synchronize region updates with a database. - -- **[How to Safely Modify the Cache from an Event Handler Callback](writing_callbacks_that_modify_the_cache.html)** - - Event handlers are synchronous. If you need to change the cache or perform any other distributed operation from event handler callbacks, be careful to avoid activities that might block and affect your overall system performance. - -- **[Cache Event Handler Examples](cache_event_handler_examples.html)** - - Some examples of cache event handlers. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/filtering_multisite_events.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/filtering_multisite_events.html.md.erb b/developing/events/filtering_multisite_events.html.md.erb deleted file mode 100644 index a46d18e..0000000 --- a/developing/events/filtering_multisite_events.html.md.erb +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: Filtering Events for Multi-Site (WAN) Distribution ---- - -You can optionally create gateway sender and/or gateway receiver filters to control which events are queued and distributed to a remote site, or to modify the data stream that is transmitted between Geode sites. - -You can implement and deploy two different types of filter for multi-site events: - -- `GatewayEventFilter`. A `GatewayEventFilter` implementation determines whether a region event is placed in a gateway sender queue and/or whether an event in a gateway queue is distributed to a remote site. You can optionally add one or more `GatewayEventFilter` implementations to a gateway sender, etiher in the `cache.xml` configuration file or using the Java API. - - Geode makes a synchronous call to the filter's `beforeEnqueue` method before it places a region event in the gateway sender queue. The filter returns a boolean value that specifies whether the event should be added to the queue. - - Geode asynchronously calls the filter's `beforeTransmit` method to determine whether the gateway sender dispatcher thread should distribute the event to a remote gateway receiver. - - For events that are distributed to another site, Geode calls the listener's `afterAcknowledgement` method to indicate that is has received an ack from the remote site after the event was received. - -- GatewayTransportFilter. Use a `GatewayTransportFilter` implementation to process the TCP stream that sends a batch of events that is distributed from one Geode cluster to another over a WAN. A `GatewayTransportFilter` is typically used to perform encryption or compression on the data that distributed. You install the same `GatewayTransportFilter` implementation on both a gateway sender and gateway receiver. - - When a gateway sender processes a batch of events for distribution, Geode delivers the stream to the `getInputStream` method of a configured `GatewayTransportFilter` implementation. The filter processes and returns the stream, which is then transmitted to the gateway receiver. When the gateway receiver receives the batch, Geode calls the `getOutputStream` method of a configured filter, which again processes and returns the stream so that the events can be applied in the local cluster. - -## <a id="topic_E97BB68748F14987916CD1A50E4B4542__section_E20B4A8A98FD4EDAAA8C14B8059AA7F7" class="no-quick-link"></a>Configuring Multi-Site Event Filters - -You install a `GatewayEventFilter` implementation to a configured gateway sender in order to decide which events are queued and distributed. You install a `GatewayTransportFilter` implementation to both a gateway sender and a gateway receiver to process the stream of batched events that are distributed between two sites: - -- **XML example** - - ``` pre - <cache> - <gateway-sender id="remoteA" parallel="true" remote-distributed-system-id="1"> - <gateway-event-filter> - <class-name>org.apache.geode.util.SampleEventFilter</class-name> - <parameter name="param1"> - <string>"value1"</string> - </parameter> - </gateway-event-filter> - <gateway-transport-filter> - <class-name>org.apache.geode.util.SampleTransportFilter</class-name> - <parameter name="param1"> - <string>"value1"</string> - </parameter> - </gateway-transport-filter> - </gateway-sender> - </cache> - ``` - - ``` pre - <cache> - ... - <gateway-receiver start-port="1530" end-port="1551"> - <gateway-transport-filter> - <class-name>org.apache.geode.util.SampleTransportFilter</class-name> - <parameter name="param1"> - <string>"value1"</string> - </parameter> - </gateway-transport-filter> - </gateway-receiver> - </cache> - ``` - -- **gfsh example** - - ``` pre - gfsh>create gateway-sender --id=remoteA --parallel=true --remote-distributed-id="1" - --gateway-event-filter=org.apache.geode.util.SampleEventFilter - --gateway-transport-filter=org.apache.geode.util.SampleTransportFilter - ``` - - See [create gateway-sender](../../tools_modules/gfsh/command-pages/create.html#topic_hg2_bjz_ck). - - ``` pre - gfsh>create gateway-receiver --start-port=1530 --end-port=1551 \ - --gateway-transport-filter=org.apache.geode.util.SampleTransportFilter - ``` - - **Note:** - You cannot specify parameters and values for the Java class you specify with the `--gateway-transport-filter` option. - - See [create gateway-receiver](../../tools_modules/gfsh/command-pages/create.html#topic_a4x_pb1_dk). - -- **API example** - - ``` pre - Cache cache = new CacheFactory().create(); - - GatewayEventFilter efilter = new SampleEventFilter(); - GatewayTransportFilter tfilter = new SampleTransportFilter(); - - GatewaySenderFactory gateway = cache.createGatewaySenderFactory(); - gateway.setParallel(true); - gateway.addGatewayEventFilter(efilter); - gateway.addTransportFilter(tfilter); - GatewaySender sender = gateway.create("remoteA", "1"); - sender.start(); - ``` - - ``` pre - Cache cache = new CacheFactory().create(); - - GatewayTransportFilter tfilter = new SampleTransportFilter(); - - GatewayReceiverFactory gateway = cache.createGatewayReceiverFactory(); - gateway.setStartPort(1530); - gateway.setEndPort(1551); - gateway.addTransportFilter(tfilter); - GatewayReceiver receiver = gateway.create(); - receiver.start(); - ``` - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/ha_event_messaging_whats_next.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/ha_event_messaging_whats_next.html.md.erb b/developing/events/ha_event_messaging_whats_next.html.md.erb deleted file mode 100644 index f61aba6..0000000 --- a/developing/events/ha_event_messaging_whats_next.html.md.erb +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: Highly Available Client/Server Event Messaging ---- - -<a id="ha_event_messaging_whats_next__section_F163917311E8478399D1DD273E8BCDF5"></a> -With server redundancy, each pool has a primary server and some number of secondaries. The primaries and secondaries are assigned on a per-pool basis and are generally spread out for load balancing, so a single client with multiple pools may have primary queues in more than one server. - -The primary server pushes events to clients and the secondaries maintain queue backups. If the primary server fails, one of the secondaries becomes primary to provide uninterrupted event messaging. - -For example, if there are six servers running and `subscription-redundancy` is set to two, one server is the primary, two servers are secondary, and the remaining three do not actively participate in HA for the client. If the primary server fails, the system assigns one of the secondaries as the new primary and attempts to add another server to the secondary pool to retain the initial redundancy level. If no new secondary server is found, then the redundancy level is not satisfied but the failover procedure completes successfully. As soon as another secondary is available, it is added. - -When high availability is enabled: - -- The primary server sends event messages to the clients. -- Periodically, the clients send received messages to the server and the server removes the sent messages from its queues. -- Periodically, the primary server synchronizes with its secondaries, notifying them of messages that can be discarded because they have already been sent and received. There is a lag in notification, so the secondary servers remain only roughly synchronized with the primary. Secondary queues contain all messages that are contained in the primary queue plus possibly a few messages that have already been sent to clients. -- In the case of primary server failure, one of the secondaries becomes the primary and begins sending event messages from its queues to the clients. Immediately after failover, the new primary usually resends some messages that were already sent by the old primary. The client recognizes these as duplicates and discards them. - -In stage 1 of this figure, the primary sends an event message to the client and a synchronization message to its secondary. By stage 2, the secondary and client have updated their queue and message tracking information. If the primary failed at stage two, the secondary would start sending event messages from its queue beginning with message A10. The client would discard the resend of message A10 and then process subsequent messages as usual. -<img src="../../images/ClientServerAdvancedTopics-5.gif" alt="High Availability Messaging: Server to Client and Primary Server to Secondary Server" id="ha_event_messaging_whats_next__image_8947A42EDEF74911BAB55B79ED8DA984" class="image" /> - -## <a id="ha_event_messaging_whats_next__section_741052B413F24F47A14F5B7D7955F0AA" class="no-quick-link"></a>Change Server Queue Synchronization Frequency - -By default, the primary server sends queue synchronization messages to the secondaries every second. You can change this interval with the `gfsh alter runtime` command - -Set the interval for queue synchronization messages as follows: - -- gfsh: - - ``` pre - gfsh>alter runtime --message-sync-interval=2 - ``` - -- XML: - - ``` pre - <!-- Set sync interval to 2 seconds --> - <cache ... message-sync-interval="2" /> - ``` - -- Java: - - ``` pre - cache = CacheFactory.create(); - cache.setMessageSyncInterval(2); - ``` - -The ideal setting for this interval depends in large part on your application behavior. These are the benefits of shorter and longer interval settings: - -- A shorter interval requires less memory in the secondary servers because it reduces queue buildup between synchronizations. In addition, fewer old messages in the secondary queues means reduced message re-sends after a failover. These considerations are most important for systems with high data update rates. -- A longer interval requires fewer distribution messages between the primary and secondary, which benefits overall system performance. - -## <a id="ha_event_messaging_whats_next__section_DF51950D30154A58818F6AD777BB3090" class="no-quick-link"></a>Set Frequency of Orphan Removal from the Secondary Queues - -Usually, all event messages are removed from secondary subscription queues based on the primary's synchronization messages. Occasionally, however, some messages are orphaned in the secondary queues. For example, if a primary fails in the middle of sending a synchronization message to its secondaries, some secondaries might receive the message and some might not. If the failover goes to a secondary that did receive the message, the system will have secondary queues holding messages that are no longer in the primary queue. The new primary will never synchronize on these messages, leaving them orphaned in the secondary queues. - -To make sure these messages are eventually removed, the secondaries expire all messages that have been enqueued longer than the time indicated by the servers' `message-time-to-live`. - -Set the time-to-live as follows: - -- XML: - - ``` pre - <!-- Set message ttl to 5 minutes --> - <cache-server port="41414" message-time-to-live="300" /> - ``` - -- Java: - - ``` pre - Cache cache = ...; - CacheServer cacheServer = cache.addCacheServer(); - cacheServer.setPort(41414); - cacheServer.setMessageTimeToLive(200); - cacheServer.start(); - ``` - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/how_cache_events_work.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/how_cache_events_work.html.md.erb b/developing/events/how_cache_events_work.html.md.erb deleted file mode 100644 index a0fa79e..0000000 --- a/developing/events/how_cache_events_work.html.md.erb +++ /dev/null @@ -1,54 +0,0 @@ ---- -title: Peer-to-Peer Event Distribution ---- - -When a region or entry operation is performed, Geode distributes the associated events in the distributed system according to system and cache configurations. - -<a id="how_cache_events_work__section_7864A275FDB549FD8E2D046DD59CB9F4"></a> -Install a cache listener for a region in each system member that needs to receive notification of region and entry changes. - -## <a id="how_cache_events_work__section_CACE500A00214CD88CE232D22899263B" class="no-quick-link"></a>Events in a Partitioned Region - -A distributed operation follows this sequence in a partitioned region: - -1. Apply the operation to the cache with the primary data entry, if appropriate. -2. Do the distribution based on the subscription-attributes interest-policy of the other members. -3. Invoke any listeners in the caches that receive the distribution. -4. Invoke the listener in the cache with the primary data entry. - -In the following figure: - -1. An API call in member M1 creates an entry. -2. The partitioned region creates the new entry in the cache in M2. M2, the holder of the primary copy, drives the rest of the procedure. -3. These two operations occur simultaneously: - - The partitioned region creates a secondary copy of the entry in the cache in M3. Creating the secondary copy does not invoke the listener on M3. - - M2 distributes the event to M4. This distribution to the other members is based on their interest policies. M4 has an interest-policy of all , so it receives notification of all events anywhere in the region. Since M1 and M3 have an interest-policy of cache-content , and this event does not affect any pre-existing entry in their local caches, they do not receive the event. - -4. The cache listener on M4 handles the notification of the remote event on M2. -5. Once everything on the other members has completed successfully, the original create operation on M2 succeeds and invokes the cache listener on M2. - -<img src="../../images/Events-2.gif" id="how_cache_events_work__image_E5E187C14A774144B85FA7B636239DBE" class="image" /> - -## <a id="how_cache_events_work__section_FACF58272C824907BA020B1727427D7A" class="no-quick-link"></a>Events in a Distributed Region - -A distributed operation follows this sequence in a distributed region: - -1. Apply the operation to the local cache, if appropriate. -2. Invoke the local listeners. -3. Do the distribution. -4. Each member that receives the distribution carries out its own operation in response, which invokes any local listeners. - -In the following figure: - -1. An entry is created through a direct API call on member M1. -2. The create invokes the cache listener on M1. -3. M1 distributes the event to the other members. -4. M2 and M3 apply the remote change through their own local operations. -5. M3 does a create, but M2 does an update, because the entry already existed in its cache. -6. The cache listener on M2 receives callbacks for the local update. Since there is no cache listener on M3, the callbacks from the create on M3 are not handled. An API call in member M1 creates an entry. - -<img src="../../images/Events-3.gif" id="how_cache_events_work__image_A24D6182B2A840D1843EBD4686966EEF" class="image" /> - -## <a id="how_cache_events_work__section_B4DCA51DDF7F44699E7355277172BEF0" class="no-quick-link"></a>Managing Events in Multi-threaded Applications - -For partitioned regions, Geode guarantees ordering of events across threads, but for distributed regions it doesnât. For multi-threaded applications that create distributed regions, you need to use your application synchronization to make sure that one operation completes before the next one begins. Distribution through the distributed-no-ack queue can work with multiple threads if you set the `conserve-sockets` attribute to true. Then the threads share one queue, preserving the order of the events in distributed regions. Different threads can invoke the same listener, so if you allow different threads to send events, it can result in concurrent invocations of the listener. This is an issue only if the threads have some shared state - if they are incrementing a serial number, for example, or adding their events to a log queue. Then you need to make your code thread safe. http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/how_client_server_distribution_works.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/how_client_server_distribution_works.html.md.erb b/developing/events/how_client_server_distribution_works.html.md.erb deleted file mode 100644 index 869adf0..0000000 --- a/developing/events/how_client_server_distribution_works.html.md.erb +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: Client-to-Server Event Distribution ---- - -Clients and servers distribute events according to client activities and according to interest registered by the client in server-side cache changes. - -<a id="how_client_server_distribution_works__section_F1070B06B3344D1CA7309934FCE097B9"></a> -When the client updates its cache, changes to client regions are automatically forwarded to the server side. The server-side update is then propagated to the other clients that are connected and have subscriptions enabled. The server does not return the update to the sending client. - -The update is passed to the server and then passed, with the value, to every other client that has registered interest in the entry key. This figure shows how a clientâs entry updates are propagated. - -<img src="../../images_svg/client_server_event_dist.svg" id="how_client_server_distribution_works__image_66AB57EEDC154962B32F7951667F4656" class="image" /> - -The figure shows the following process: - -1. Entry X is updated or created in Region A through a direct API call on Client1. -2. The update to the region is passed to the pool named in the region. -3. The pool propagates the event to the cache server, where the region is updated. -4. The server member distributes the event to its peers and also places it into the subscription queue for Client2 because that client has previously registered interest in entry X. -5. The event for entry X is sent out of the queue to Client2. When this happens is indeterminate. - -Client to server distribution uses the client pool connections to send updates to the server. Any region with a named pool automatically forwards updates to the server. Client cache modifications pass first through a client `CacheWriter`, if one is defined, then to the server through the client pool, and then finally to the client cache itself. A cache writer, either on the client or server side, may abort the operation. - -| Change in Client Cache | Effect on Server Cache | -|-----------------------------------------------------|-----------------------------------------------------------------------------------------------------------| -| Entry create or update | Creation or update of entry. | -| Distributed entry destroy | Entry destroy. The destroy call is propagated to the server even if the entry is not in the client cache. | -| Distributed region destroy/clear (distributed only) | Region destroy/clear | - -**Note:** -Invalidations on the client side are not forwarded to the server. - -## <a id="how_client_server_distribution_works__section_A16562611E094C88B12BC149D5EEEEBA" class="no-quick-link"></a>Server-to-Client Event Distribution - -The server automatically sends entry modification events only for keys in which the client has registered interest. In the interest registration, the client indicates whether to send new values or just invalidations for the server-side entry creates and updates. If invalidation is used, the client then updates the values lazily as needed. - -This figure shows the complete event subscription event distribution for interest registrations, with value receipt requested (receiveValues=true) and without. - -<img src="../../images_svg/server_client_event_dist.svg" id="how_client_server_distribution_works__image_7FD1450B9D58429F860400801EDFDCAE" class="image" /> - -<table> -<colgroup> -<col width="50%" /> -<col width="50%" /> -</colgroup> -<thead> -<tr class="header"> -<th>Change in Server Cache</th> -<th>Effect on Client Cache</th> -</tr> -</thead> -<tbody> -<tr class="odd"> -<td>Entry create/update</td> -<td>For subscriptions with <code class="ph codeph">receiveValues</code> set to true, entry create or update. -<p></p> -For subscriptions with <code class="ph codeph">receiveValues</code> set to false, entry invalidate if the entry already exists in the client cache; otherwise, no effect. The next client get for the entry is forwarded to the server.</td> -</tr> -<tr class="even"> -<td>Entry invalidate/destroy (distributed only)</td> -<td>Entry invalidate/destroy</td> -</tr> -<tr class="odd"> -<td>Region destroy/clear (distributed only)</td> -<td>Region destroy or local region clear</td> -</tr> -</tbody> -</table> - -Server-side distributed operations are all operations that originate as a distributed operation in the server or one of its peers. Region invalidation in the server is not forwarded to the client. - -**Note:** -To maintain a unified set of data in your servers, do not do local entry invalidation in your server regions. - -## <a id="how_client_server_distribution_works__section_34613292788D4FA4AB730045FB9A405A" class="no-quick-link"></a>Server-to-Client Message Tracking - -The server uses an asynchronous messaging queue to send events to its clients. Every event in the queue originates in an operation performed by a thread in a client, a server, or an application in the serverâs or some other distributed system. The event message has a unique identifier composed of the originating threadâs ID combined with its memberâs distributed system member ID, and the sequential ID of the operation. So the event messages originating in any single thread can be grouped and ordered by time from lowest sequence ID to highest. Servers and clients track the highest sequential ID for each member thread ID. - -A single client thread receives and processes messages from the server, tracking received messages to make sure it does not process duplicate sends. It does this using the process IDs from originating threads. - -<img src="../../images_svg/client_server_message_tracking.svg" id="how_client_server_distribution_works__image_F4F9D13252E14F11AD63240AED39191A" class="image" /> - -The clientâs message tracking list holds the highest sequence ID of any message received for each originating thread. The list can become quite large in systems where there are many different threads coming and going and doing work on the cache. After a thread dies, its tracking entry is not needed. To avoid maintaining tracking information for threads that have died, the client expires entries that have had no activity for more than the `subscription-message-tracking-timeout`. - -## <a id="how_client_server_distribution_works__section_99E436C569F3422AA842AA74F73A6B36" class="no-quick-link"></a>Client Interest Registration on the Server - -The system processes client interest registration following these steps: - -1. The entries in the client region that may be affected by this registration are silently destroyed. Other keys are left alone. - - For the `registerInterest` method, the system destroys all of the specified keys, leaving other keys in the client region alone. So if you have a client region with keys A, B, and C and you register interest in the key list A, B, at the start of the `registerInterest` operation, the system destroys keys A and B in the client cache but does not touch key C. - - For the `registerInterestRegex` method, the system silently destroys all keys in the client region. - -2. The interest specification is sent to the server, where it is added to the clientâs interest list. The list can specify entries that are not in the server region at the time interest is registered. -3. If a bulk load is requested in the call's `InterestResultPolicy` parameter, before control is returned to the calling method, the server sends all data that currently satisfies the interest specification. The client's region is updated automatically with the downloaded data. If the server region is partitioned, the entire partitioned region is used in the bulk load. Otherwise, only the serverâs local cache region is used. The interest results policy options are: - - KEYSâThe client receives a bulk load of all available keys matching the interest registration criteria. - - KEYS\_VALUESâThe client receives a bulk load of all available keys and values matching the interest registration criteria. This is the default interest result policy. - - NONEâThe client does not receive any immediate bulk loading. - -Once interest is registered, the server continually monitors region activities and sends events to its clients that match the interest. - -- No events are generated by the register interest calls, even if they load values into the client cache. -- The server maintains the union of all of the interest registrations, so if a client registers interest in key âAâ, then registers interest in regular expression "B\*", the server will send updates for all entries with key âAâ or key beginning with the letter âBâ. -- The server maintains the interest registration list separate from the region. The list can contain specifications for entries that are not currently in the server region. -- The `registerInterestRegex` method uses the standard `java.util.regex` methods to parse the key specification. - -## <a id="how_client_server_distribution_works__section_928BB60066414BEB9FAA7FB3120334A3" class="no-quick-link"></a>Server Failover - -When a server hosting a subscription queue fails, the queueing responsibilities pass to another server. How this happens depends on whether the new server is a secondary server. In any case, all failover activities are carried out automatically by the Geode system. - -- **Non-HA failover:** The client fails over without high availability if it is not configured for redundancy or if all secondaries also fail before new secondaries can be initialized. As soon as it can attach to a server, the client goes through an automatic reinitialization process. In this process, the failover code on the client side silently destroys all entries of interest to the client and refetches them from the new server, essentially reinitializing the client cache from the new serverâs cache. For the notify all configuration, this clears and reloads all of the entries for the client regions that are connected to the server. For notify by subscription, it clears and reloads only the entries in the region interest lists. To reduce failover noise, the events caused by the local entry destruction and refetching are blocked by the failover code and do not reach the client cache listeners. Because of this, your clients could receive some out-of-sequence events during and af ter a server failover. For example, entries that exist on the failed server and not on its replacement are destroyed and never recreated during a failover. Because the destruction events are blocked, the client ends up with entries removed from its cache with no associated destroy events. -- **HA failover:** If your client pool is configured with redundancy and a secondary server is available at the time the primary fails, the failover is invisible to the client. The secondary server resumes queueing activities as soon as the primary loss is detected. The secondary might resend a few events, which are discarded automatically by the client message tracking activities. - - **Note:** - There is a very small potential for message loss during HA server failover. The risk is not present for failover to secondaries that have fully initialized their subscription queue data. The risk is extremely low in healthy systems that use at least two secondary servers. The risk is higher in unstable systems where servers often fail and where secondaries do not have time to initialize their subscription queue data before becoming primaries. To minimize the risk, the failover logic chooses the longest-lived secondary as the new primary. - - - **Note:** - Redundancy management is handled by the client, so when a durable client is disconnected from the server, client event redundancy is not maintained. Even if the servers fail one at a time, so that running clients have time to fail over and pick new secondary servers, an offline durable client cannot fail over. As a result, the client loses its queued messages. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/how_events_work.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/how_events_work.html.md.erb b/developing/events/how_events_work.html.md.erb deleted file mode 100644 index 4553582..0000000 --- a/developing/events/how_events_work.html.md.erb +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: How Events Work ---- - -Members in your Geode distributed system receive cache updates from other members through cache events. The other members can be peers to the member, clients or servers or other distributed systems. - -## <a id="how_events_work__section_6C75098DDBB84944ADE57F2088330D5A" class="no-quick-link"></a>Events Features - -These are the primary features of Geode events: - -- Content-based events -- Asynchronous event notifications with conflation -- Synchronous event notifications for low latency -- High availability through redundant messaging queues -- Event ordering and once and only-once delivery -- Distributed event notifications -- Durable subscriptions -- Continuous querying - -## <a id="how_events_work__section_F5F0E506652940DFBA0D6B7AAA72E3EF" class="no-quick-link"></a>Types of Events - -There are two categories of events and event handlers. - -- Cache events in the caching API are used by applications with a cache. Cache events provide detail-level notification for changes to your data. Continuous query events are in this category. -- Administrative events in the administration API are used by administrative applications without caches. - -Both kinds of events can be generated by a single member operation. - -**Note:** -You can handle one of these categories of events in a single system member. You cannot handle both cache and administrative events in a single member. - -Because Geode maintains the order of administrative events and the order of cache events separately, using cache events and administrative events in a single process can cause unexpected results. - -## <a id="how_events_work__section_4BCDB22AB927478EBF1035B0DE230DD3" class="no-quick-link"></a>Event Cycle - -The following steps describe the event cycle: - -1. An operation begins, such as data put or a cache close. -2. The operation execution generates these objects: - - An object of type `Operation` that describes the method that triggered the event. - - An event object that describes the event, such as the member and region where the operation originated. - -3. The event handlers that can handle the event are called and passed the event objects. Different event types require different handler types in different locations. If there is no matching event handler, that does not change the effect of the operation, which happens as usual. -4. When the handler receives the event, it triggers the handlerâs callback method for this event. The callback method can hand off the event object as input to another method. Depending on the type of event handler, the callbacks can be triggered before or after the operation. The timing depends on the event handler, not on the event itself. - **Note:** - For transactions, after-operation listeners receive the events after the transaction has committed. - -5. If the operation is distributed, so that it causes follow-on operations in other members, those operations generate their own events, which can be handled by their listeners in the same way. - -## <a id="how_events_work__section_E48ECC8A1B39411AA23D17BA0C05517E" class="no-quick-link"></a>Event Objects - -Event objects come in several types, depending on the operation. Some operations generate multiple objects of different types. All event objects contain data describing the event, and each event type carries slightly different kinds of data appropriate to its matching operation. An event object is stable. For example, its content does not change if you pass it off to a method on another thread. - -For cache events, the event object describes the operation performed in the local cache. If the event originated remotely, it describes the local application of the remote entry operation, not the remote operation itself. The only exception is when the local region has an empty data policy; then the event carries the information for the remote (originating) cache operation. - -## <a id="how_events_work__section_2EA59E9F7203433A8AD248C499D61BF4" class="no-quick-link"></a>Event Distribution - -After a member processes an event in its local cache, it distributes it to remote caches according to the member's configuration and the configurations of the remote caches. For example, if a client updates its cache, the update is forwarded to the client's server. The server distributes the update to its peers and forwards it to any other clients according to their interest in the data entry. If the server system is part of a multi-site deployment and the data region is configured to use a gateway sender, then the gateway sender also forwards the update to a remote site, where the update is further distributed and propagated. - -## <a id="how_events_work__section_C18D3CA923FB427AA01DD811589D63C0" class="no-quick-link"></a>Event Handlers and Region Data Storage - -You can configure a region for no local data storage and still send and receive events for the region. Conversely, if you store data in the region, the cache is updated with data from the event regardless of whether you have any event handlers installed. - -## <a id="how_events_work__section_22EB4B9E6C4445F898DB64A769780460" class="no-quick-link"></a>Multiple Listeners - -When multiple listeners are installed, as can be done with cache listeners, the listeners are invoked sequentially in the order they were added to the region or cache. Listeners are executed one at a time. So, unless you program a listener to pass off processing to another thread, you can use one listener's work in later listeners. - -## <a id="how_events_work__section_C4758D7E2CA2498A87315DE903A07AE4" class="no-quick-link"></a>Event Ordering - -During a cache operation, event handlers are called at various stages of the operation. Some event handlers are called before a region update and some are called after the region update operation. Depending on the type of event handler being called, the event handler can receive the events in-order or out-of-order in which they are applied on Region. - -- `CacheWriter` and `AsyncEventListener` always receive events in the order in which they are applied on region. -- `CacheListener` and `CqListener` can receive events in a different order than the order in which they were applied on the region. - -**Note:** -An `EntryEvent` contains both the old value and the new value of the entry, which helps to indicate the value that was replaced by the cache operation on a particular key. - -- **[Peer-to-Peer Event Distribution](../../developing/events/how_cache_events_work.html)** - - When a region or entry operation is performed, Geode distributes the associated events in the distributed system according to system and cache configurations. - -- **[Client-to-Server Event Distribution](../../developing/events/how_client_server_distribution_works.html)** - - Clients and servers distribute events according to client activities and according to interest registered by the client in server-side cache changes. - -- **[Multi-Site (WAN) Event Distribution](../../developing/events/how_multisite_distribution_works.html)** - - Geode distributes a subset of cache events between distributed systems, with a minimum impact on each system's performance. Events are distributed only for regions that you configure to use a gateway sender for distribution. - -- **[List of Event Handlers and Events](../../developing/events/list_of_event_handlers_and_events.html)** - - Geode provides many types of events and event handlers to help you manage your different data and application needs. - - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/how_multisite_distribution_works.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/how_multisite_distribution_works.html.md.erb b/developing/events/how_multisite_distribution_works.html.md.erb deleted file mode 100644 index c43d850..0000000 --- a/developing/events/how_multisite_distribution_works.html.md.erb +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: Multi-Site (WAN) Event Distribution ---- - -Geode distributes a subset of cache events between distributed systems, with a minimum impact on each system's performance. Events are distributed only for regions that you configure to use a gateway sender for distribution. - -## <a id="how_multisite_distribution_works__section_A16562611E094C88B12BC149D5EEEEBA" class="no-quick-link"></a>Queuing Events for Distribution - -In regions that are configured with one or more gateway senders (`gateway-sender-ids` attribute), events are automatically added to a gateway sender queue for distribution to other sites. Events that are placed in a gateway sender queue are distributed asynchronously to remote sites. For serial gateway queues, the ordering of events sent between sites can be preserved using the `order-policy` attribute. - -If a queue becomes too full, it is overflowed to disk to keep the member from running out of memory. You can optionally configure the queue to be persisted to disk (with the `enable-persistence` `gateway-sender` attribute). With persistence, if the member that manages the queue goes down, the member picks up where it left off after it restarts. - -## Operation Distribution from a Gateway Sender - -The multi-site installation is designed for minimal impact on distributed system performance, so only the farthest-reaching entry operations are distributed between sites. - -These operations are distributed: - -- entry create -- entry put -- entry distributed destroy, providing the operation is not an expiration action - -These operations are not distributed: - -- get -- invalidate -- local destroy -- expiration actions of any kind -- region operations - -## <a id="how_multisite_distribution_works__section_EE819CBF41274312BD5C3EA4A660475C" class="no-quick-link"></a>How a Gateway Sender Processes Its Queue - -Each primary gateway sender contains a processor thread that reads messages from the queue, batches them, and distributes the batches to a gateway receiver in a remote site. To process the queue, a gateway sender thread takes the following actions: - -1. Reads messages from the queue -2. Creates a batch of the messages -3. Synchronously distributes the batch to the other site and waits for a reply -4. Removes the batch from the queue after the other site has successfully replied - -Because the batch is not removed from the queue until after the other site has replied, the message cannot get lost. On the other hand, in this mode a message could be processed more than once. If a site goes offline in the middle of processing a batch of messages, then that same batch will be sent again once the site is back online. - -You can configure the batch size for messages as well as the batch time interval settings. A gateway sender processes a batch of messages from the queue when either the batch size or the time interval is reached. In an active network, it is likely that the batch size will be reached before the time interval. In an idle network, the time interval will most likely be reached before the batch size. This may result in some network latency that corresponds to the time interval. - -## <a id="how_multisite_distribution_works__section_EF240AB26CF242F99689222E9E1D2512" class="no-quick-link"></a>How a Gateway Sender Handles Batch Processing Failure - -Exceptions can occur at different points during batch processing: - -- The gateway receiver could fail with acknowledgment. If processing fails while the gateway receiver is processing a batch, the receiver replies with a failure acknowledgment that contains the exception, including the identity of the message that failed, and the ID of the last message that it successfully processed. The gateway sender then removes the successfully processed messages and the failed message from the queue and logs an exception with the failed message information. The sender then continues processing the messages remaining in the queue. -- The gateway receiver can fail without acknowledgment. If the gateway receiver does not acknowledge a sent batch, the gateway sender does not know which messages were successfully processed. In this case the gateway sender re-sends the entire batch. -- No gateway receivers may be available for processing. If a batch processing exception occurs because there are no remote gateway receivers available, then the batch remains in the queue. The gateway sender waits for a time, and then attempts to re-send the batch. The time period between attempts is five seconds. The existing server monitor continuously attempts to connect to the gateway receiver, so that a connection can be made and queue processing can continue. Messages build up in the queue and possibly overflow to disk while waiting for the connection. - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/implementing_cache_event_handlers.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/implementing_cache_event_handlers.html.md.erb b/developing/events/implementing_cache_event_handlers.html.md.erb deleted file mode 100644 index 583fcdc..0000000 --- a/developing/events/implementing_cache_event_handlers.html.md.erb +++ /dev/null @@ -1,136 +0,0 @@ ---- -title: Implementing Cache Event Handlers ---- - -Depending on your installation and configuration, cache events can come from local operations, peers, servers, and remote sites. Event handlers register their interest in one or more events and are notified when the events occur. - -<a id="implementing_cache_event_handlers__section_9286E8C6B3C54089888E1680B4F43692"></a> -For each type of handler, Geode provides a convenience class with empty stubs for the interface callback methods. - -**Note:** -Write-behind cache listeners are created by extending the `AsyncEventListener` interface, and they are configured with an `AsyncEventQueue` that you assign to one or more regions. - -**Procedure** - -1. Decide which events your application needs to handle. For each region, decide which events you want to handle. For the cache, decide whether to handle transaction events. -2. For each event, decide which handlers to use. The `*Listener` and `*Adapter` classes in `org.apache.geode.cache.util` show the options. -3. Program each event handler: - - 1. Extend the handler's adapter class. - 2. If you want to declare the handler in the `cache.xml`, implement the `org.apache.geode.cache.Declarable` interface as well. - 3. Implement the handler's callback methods as needed by your application. - - **Note:** - Improperly programmed event handlers can block your distributed system. Cache events are synchronous. To modify your cache or perform distributed operations based on events, avoid blocking your system by following the guidelines in [How to Safely Modify the Cache from an Event Handler Callback](writing_callbacks_that_modify_the_cache.html#writing_callbacks_that_modify_the_cache). - - Example: - - ``` pre - package myPackage; - import org.apache.geode.cache.Declarable; - import org.apache.geode.cache.EntryEvent; - import org.apache.geode.cache.util.CacheListenerAdapter; - import java.util.Properties; - - public class MyCacheListener extends CacheListenerAdapter implements Declarable { - /** Processes an afterCreate event. - * @param event The afterCreate EntryEvent received - */ - public void afterCreate(EntryEvent event) { - String eKey = event.getKey(); - String eVal = event.getNewValue(); - ... do work with event info - } - ... process other event types - } - - ``` - -4. Install the event handlers, either through the API or the `cache.xml`. - - XML Region Event Handler Installation: - - ``` pre - <region name="trades"> - <region-attributes ... > - <!-- Cache listener --> - <cache-listener> - <class-name>myPackage.MyCacheListener</class-name> - <cache-listener> - </region-attributes> - </region> - ``` - - Java Region Event Handler Installation: - - ``` pre - tradesRegion = cache.createRegionFactory(RegionShortcut.PARTITION) - .addCacheListener(new MyCacheListener()) - .create("trades"); - ``` - - XML Transaction Writer and Listener Installation: - - ``` pre - <cache search-timeout="60"> - <cache-transaction-manager> - <transaction-listener> - <class-name>com.company.data.MyTransactionListener</class-name> - <parameter name="URL"> - <string>jdbc:cloudscape:rmi:MyData</string> - </parameter> - </transaction-listener> - <transaction-listener> - . . . - </transaction-listener> - <transaction-writer> - <class-name>com.company.data.MyTransactionWriter</class-name> - <parameter name="URL"> - <string>jdbc:cloudscape:rmi:MyData</string> - </parameter> - <parameter - ... - </parameter> - </transaction-writer> - </cache-transaction-manager> - . . . - </cache> - ``` - -The event handlers are initialized automatically during region creation when you start the member. - -## <a id="implementing_cache_event_handlers__section_C62E9535C43B4BC5A7AA7B8B4125D1EB" class="no-quick-link"></a>Installing Multiple Listeners on a Region - -XML: - -``` pre -<region name="exampleRegion"> - <region-attributes> - . . . - <cache-listener> - <class-name>myCacheListener1</class-name> - </cache-listener> - <cache-listener> - <class-name>myCacheListener2</class-name> - </cache-listener> - <cache-listener> - <class-name>myCacheListener3</class-name> - </cache-listener> - </region-attributes> -</region> -``` - -API: - -``` pre -CacheListener listener1 = new myCacheListener1(); -CacheListener listener2 = new myCacheListener2(); -CacheListener listener3 = new myCacheListener3(); - -Region nr = cache.createRegionFactory() - .initCacheListeners(new CacheListener[] - {listener1, listener2, listener3}) - .setScope(Scope.DISTRIBUTED_NO_ACK) - .create(name); - -``` http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/implementing_durable_client_server_messaging.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/implementing_durable_client_server_messaging.html.md.erb b/developing/events/implementing_durable_client_server_messaging.html.md.erb deleted file mode 100644 index d60f55a..0000000 --- a/developing/events/implementing_durable_client_server_messaging.html.md.erb +++ /dev/null @@ -1,182 +0,0 @@ ---- -title: Implementing Durable Client/Server Messaging ---- - -<a id="implementing_durable_client_server_messaging__section_7A0D0B7D1F2748C7BA7479E0FD5C6BFA"></a> -Use durable messaging for subscriptions that you need maintained for your clients even when your clients are down or disconnected. You can configure any of your event subscriptions as durable. Events for durable queries and subscriptions are saved in queue when the client is disconnected and played back when the client reconnects. Other queries and subscriptions are removed from the queue. - -Use durable messaging for client/server installations that use event subscriptions. - -These are the high-level tasks described in this topic: - -1. Configure your client as durable -2. Decide which subscriptions should be durable and configure accordingly -3. Program your client to manage durable messaging for disconnect, reconnect, and event handling - -## <a id="implementing_durable_client_server_messaging__section_643EB5FA6F09463C80646786394F1E02" class="no-quick-link"></a>Configure the Client as Durable - -Use one of the following methods: - -- `gemfire.properties` file: - - ``` pre - durable-client-id=31 - durable-client-timeout=200 - ``` - -- Java: - - ``` pre - Properties props = new Properties(); - props.setProperty("durable-client-id", "31"); - props.setProperty("durable-client-timeout", "" + 200); - CacheFactory cf = new CacheFactory(props); - ``` - -<a id="implementing_durable_client_server_messaging__section_B8E01FE4B5A347EB96085DA3194F6AE8"></a> - -The `durable-client-id` indicates that the client is durable and gives the server an identifier to correlate the client to its durable messages. For a non-durable client, this id is an empty string. The ID can be any number that is unique among the clients attached to servers in the same distributed system. - -The `durable-client-timeout` tells the server how long to wait for client reconnect. When this timeout is reached, the server stops storing to the client's message queue and discards any stored messages. The default is 300 seconds. This is a tuning parameter. If you change it, take into account the normal activity of your application, the average size of your messages, and the level of risk you can handle, both in lost messages and in the servers' capacity to store enqueued messages. Assuming that no messages are being removed from the queue, how long can the server run before the queue reaches the maximum capacity? How many durable clients can the server handle? To assist with tuning, use the Geode message queue statistics for durable clients through the disconnect and reconnect cycles. - -## <a id="implementing_durable_client_server_messaging__section_BB5DCCE0582E4FE8B62DE473512FC704" class="no-quick-link"></a>Configure Durable Subscriptions and Continuous Queries - -The register interest and query creation methods all have an optional boolean parameter for indicating durability. By default all are non-durable. - -``` pre -// Durable registration -// Define keySpecification, interestResultPolicy, durability -exampleRegion.registerInterest(keySpecification, interestResultPolicySpecification, true); - -// Durable CQ -// Define cqName, queryString, cqAttributes, durability -CqQuery myCq = queryService.newCq(cqName, queryString, cqAttributes, true); -``` - -Save only critical messages while the client is disconnected by only indicating durability for critical subscriptions and CQs. When the client is connected to its servers, it receives messages for all keys and queries reqistered. When the client is disconnected, non-durable interest registrations and CQs are discontinued but all messages already in the queue for them remain there. - -**Note:** -For a single durable client ID, you must maintain the same durability of your registrations and queries between client runs. - -## <a id="implementing_durable_client_server_messaging__section_0FBD23CC79784E588135FE93306EC0A4" class="no-quick-link"></a>Program the Client to Manage Durable Messaging - -Program your durable client to be durable-messaging aware when it disconnects, reconnects, and handles events from the server. - -1. Disconnect with a request to keep your queues active by using `Pool.close` or `ClientCache.close` with the boolean `keepalive` parameter. - - ``` pre - clientCache.close(true); - ``` - - To be retained during client down time, durable continuous queries (CQs) must be executing at the time of disconnect. - -2. Program your durable client's reconnection to: - - 1. If desired, detect whether the previously registered subscription queue is available upon durable client reconnection and the count of pending events in the queue. Based on the results, you can then decide whether to receive the remaining events or close the cache if the number is too large. - - For example, for a client with only the default pool created: - - ``` pre - int pendingEvents = cache.getDefaultPool().getPendingEventCount(); - - if (pendingEvents == -2) { // client connected for the first time ⦠// continue - } - else if (pendingEvents == -1) { // client reconnected but after the timeout period - ⦠// handle possible data loss - } - else { // pendingEvents >= 0 - ⦠// decide to invoke readyForEvents() or ClientCache::close(false)/pool.destroy() - } - ``` - - For a client with multiple pools: - - ``` pre - int pendingEvents = 0; - - int pendingEvents1 = PoolManager.find(âpool1â).getPendingEventCount(); - - pendingEvents += (pendingEvents1 > 0) ? pendingEvents1 : 0; - - int pendingEvents2 = PoolManager.find(âpool2â).getPendingEventCount(); - - pendingEvents += (pendingEvents2 > 0) ? pendingEvents2 : 0; - - // process individual pool counts separately. - ``` - - The `getPendingEventCount` API can return the following possible values: - - A value representing a count of events pending at the server. Note that this count is an approximate value based on the time the durable client pool connected or reconnected to the server. Any number of invocations will return the same value. - - A zero value if there are no events pending at server for this client pool - - A negative value indicates that no queue is available at the server for the client pool. - - -1 indicates that the client pool has reconnected to the server after its durable-client-timeout period has elapsed. The pool's subscription queue has been removed possibly causing data loss. - - A value of -2 indicates that this client pool has connected to server for the first time. - - 2. Connect, initialize the client cache, regions, and any cache listeners, and create and execute any durable continuous queries. - 3. Run all interest registration calls. - - **Note:** - Registering interest with `InterestResultPolicy.KEYS_VALUES` initializes the client cache with the *current* values of specified keys. If concurrency checking is enabled for the region, any earlier (older) region events that are replayed to the client are ignored and are not sent to configured listeners. If your client must process all replayed events for a region, register with `InterestResultPolicy.KEYS` or `InterestResultPolicy.NONE` when reconnecting. Or, disable concurrency checking for the region in the client cache. See [Consistency for Region Updates](../distributed_regions/region_entry_versions.html#topic_CF2798D3E12647F182C2CEC4A46E2045). - - 4. Call `ClientCache.readyForEvents` so the server will replay stored events. If the ready message is sent earlier, the client may lose events. - - ``` pre - ClientCache clientCache = ClientCacheFactory.create(); - // Here, create regions, listeners that are not defined in the cache.xml . . . - // Here, run all register interest calls before doing anything else - clientCache.readyForEvents(); - ``` - -3. When you program your durable client `CacheListener`: - 1. Implement the callback methods to behave properly when stored events are replayed. The durable clientâs `CacheListener` must be able to handle having events played after the fact. Generally listeners receive events very close to when they happen, but the durable client may receive events that occurred minutes before and are not relevant to current cache state. - 2. Consider whether to use the `CacheListener` callback method, `afterRegionLive`, which is provided specifically for the end of durable event replay. You can use it to perform application-specific operations before resuming normal event handling. If you do not wish to use this callback, and your listener is an instance of `CacheListener` (instead of a `CacheListenerAdapter`) implement `afterRegionLive` as an empty method. - -## Initial Operation - -The initial startup of a durable client is similar to the startup of any other client, except that it specifically calls the `ClientCache.readyForEvents` method when all regions and listeners on the client are ready to process messages from the server. - -## <a id="implementing_durable_client_server_messaging__section_9B9A9EE8C7FF47948C8108A0F7F4E32E" class="no-quick-link"></a>Disconnection - -While the client and servers are disconnected, their operation varies depending on the circumstances. - -- **Normal disconnect**. When a client closes its connection, the servers stop sending messages to the client and release its connection. If the client requests it, the servers maintain the queues and durable interest list information until the client reconnects or times out. The non-durable interest lists are discarded. The servers continue to queue up incoming messages for entries on the durable interest list. All messages that were in the queue when the client disconnected remain in the queue. If the client requests not to have its subscriptions maintained, or if there are no durable subscriptions, the servers unregister the client and do the same cleanup as for a non-durable client. -- **Abnormal disconnect**. If the client crashes or loses its connections to all servers, the servers automatically maintain its message queue and durable subscriptions until it reconnects or times out. -- **Client disconnected but operational**. If the client operates while it is disconnected, it gets what data it can from the local client cache. Since updates are not allowed, the data can become stale. An `UnconnectedException` occurs if an update is attempted. -- **Client stays disconnected past timeout period**. The servers track how long to keep a durable subscription queue alive based on the `durable-client-timeout` setting. If the client remains disconnected longer than the timeout, the servers unregister the client and do the same cleanup that is performed for a non-durable client. The servers also log an alert. When a timed-out client reconnects, the servers treat it as a new client making its initial connection. - -## <a id="implementing_durable_client_server_messaging__section_E3C42A6FDC884FC38ECC121955C06BDC" class="no-quick-link"></a>Reconnection - -During initialization, the client cache is not blocked from doing operations, so you might be receiving old stored events from the server at the same time that your client cache is being updated by much more current events. These are the things that can act on the cache concurrently: - -- Results returned by the server in response to the clientâs interest registrations. -- Client cache operations by the application. -- Callbacks triggered by replaying old events from the queue - -Geode handles the conflicts between the application and interest registrations so they do not create cache update conflicts. But you must program your event handlers so they don't conflict with current operations. This is true for all event handlers, but it is especially important for those used in durable clients. Your handlers may receive events well after the fact and you must ensure your programming takes that into account. - -This figure shows the three concurrent procedures during the initialization process. The application begins operations immediately on the client (step 1), while the clientâs cache ready message (also step 1) triggers a series of queue operations on the servers (starting with step 2 on the primary server). At the same time, the client registers interest (step 2 on the client) and receives a response from the server. Message B2 applies to an entry in Region A, so the cache listener handles B2âs event. Because B2 comes before the marker, the client does not apply the update to the cache. - -<img src="../../images/ClientServerAdvancedTopics-6.gif" alt="Durable client reconnection. " id="implementing_durable_client_server_messaging__image_068A59FA019E46FA9DE0BC7FA60AAADD" class="image" /> - -## <a id="implementing_durable_client_server_messaging__section_C848DF6D649F4DCAA2B895F5439BAA97" class="no-quick-link"></a>Durable Event Replay - -When a durable client reconnects before the timeout period, the servers replay the events that were stored while the client was gone and then resume normal event messaging to the client. To avoid overwriting current entries with old data, the stored events are not applied to the client cache. Stored events are distinguished from new normal events by a marker that is sent to the client once all old events are replayed. - -1. All servers with a queue for this client place a marker in their queue when the client reconnects. -2. The primary server sends the queued messages to the client up to the marker. -3. The client receives the messages but does not apply the usual automatic updates to its cache. If cache listeners are installed, they handle the events. -4. The client receives the marker message indicating that all past events have been played back. -5. The server sends the current list of live regions. -6. For every `CacheListener` in each live region on the client, the marker event triggers the `afterRegionLive` callback. After the callback, the client begins normal processing of events from the server and applies the updates to its cache. - -Even when a new client starts up for the first time, the client cache ready markers are inserted in the queues. If messages start coming into the new queues before the servers insert the marker, those messages are considered as having happened while the client was disconnected, and their events are replayed the same as in the reconnect case. - -## <a id="implementing_durable_client_server_messaging__section_E519D541E2844292ABD2E0BDF5FB5798" class="no-quick-link"></a>Application Operations During Interest Registration - -Application operations take precedence over interest registration responses. The client can perform operations while it is receiving its interest registration responses. When adding register interest responses to the client cache, the following rules are applied: - -- If the entry already exists in the cache with a valid value, it is not updated. -- If the entry is invalid, and the register interest response is valid, the valid value is put into the cache. -- If an entry is marked destroyed, it is not updated. Destroyed entries are removed from the system after the register interest response is completed. -- If the interest response does not contain any results, because all of those keys are absent from the serverâs cache, the clientâs cache can start out empty. If the queue contains old messages related to those keys, the events are still replayed in the clientâs cache. - http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/developing/events/implementing_write_behind_event_handler.html.md.erb ---------------------------------------------------------------------- diff --git a/developing/events/implementing_write_behind_event_handler.html.md.erb b/developing/events/implementing_write_behind_event_handler.html.md.erb deleted file mode 100644 index 908b0f5..0000000 --- a/developing/events/implementing_write_behind_event_handler.html.md.erb +++ /dev/null @@ -1,228 +0,0 @@ ---- -title: Implementing an AsyncEventListener for Write-Behind Cache Event Handling ---- - -An `AsyncEventListener` asynchronously processes batches of events after they have been applied to a region. You can use an `AsyncEventListener` implementation as a write-behind cache event handler to synchronize region updates with a database. - -## <a id="implementing_write_behind_cache_event_handling__section_35B3ADC77E1147468A568E49C8C308E1" class="no-quick-link"></a>How an AsyncEventListener Works - -An `AsyncEventListener` instance is serviced by its own dedicated thread in which a callback method is invoked. Events that update a region are placed in an internal `AsyncEventQueue`, and one or more threads dispatch batches of events at a time to the listener implementation. - -You can configure an `AsyncEventQueue` to be either serial or parallel. A serial queue is deployed to one Geode member, and it delivers all of a region's events, in order of occurrence, to a configured `AsyncEventListener` implementation. A parallel queue is deployed to multiple Geode members, and each instance of the queue delivers region events, possibly simultaneously, to a local `AsyncEventListener` implementation. - -While a parallel queue provides the best throughput for writing events, it provides less control for ordering those events. With a parallel queue, you cannot preserve event ordering for a region as a whole because multiple Geode servers queue and deliver the region's events at the same time. However, the ordering of events for a given partition (or for a given queue of a distributed region) can be preserved. - -For both serial and parallel queues, you can control the maximum amount of memory that each queue uses, as well as the batch size and frequency for processing batches in the queue. You can also configure queues to persist to disk (instead of simply overflowing to disk) so that write-behind caching can pick up where it left off when a member shuts down and is later restarted. - -Optionally, a queue can use multiple threads to dispatch queued events. When you configure multiple threads for a serial queue, the logical queue that is hosted on a Geode member is divided into multiple physical queues, each with a dedicated dispatcher thread. You can then configure whether the threads dispatch queued events by key, by thread, or in the same order in which events were added to the queue. When you configure multiple threads for a parallel queue, each queue hosted on a Geode member is processed by dispatcher threads; the total number of queues created depends on the number of members that host the region. - -A `GatewayEventFilter` can be placed on the `AsyncEventQueue` to control whether a particular event is sent to a selected `AsyncEventListener`. For example, events associated with sensitive data could be detected and not queued. For more detail, see the Javadocs for `GatewayEventFilter`. - -A `GatewayEventSubstitutionFilter` can specify whether the event is transmitted in its entirety or in an altered representation. For example, to reduce the size of the data being serialized, it might be a more efficient to represent a full object by only its key. For more detail, see the Javadocs for `GatewayEventSubstitutionFilter`. - -## Operation Distribution from an AsyncEventQueue - -An `AsyncEventQueue` distributes these operations: - -- Entry create -- Entry put -- Entry distributed destroy, providing the operation is not an expiration action -- Expiration destroy, if the `forward-expiration-destroy` attribute is set to `true`. By default, this attribute is `false`, but you can set it to `true` using `cache.xml` or `gfsh`. To set this attribute in the Java API, use `AsyncEventQueueFactory.setForwardExpirationDestroy()`. See the javadocs for details. - -These operations are not distributed: - -- Get -- Invalidate -- Local destroy -- Region operations -- Expiration actions -- Expiration destroy, if the `forward-expiration-destroy` attribute is set to `false`. The default value is `false`. - -## <a id="implementing_write_behind_cache_event_handling__section_6FDBAFCB9C194EB0AF0822A509F2F9F2" class="no-quick-link"></a>Guidelines for Using an AsyncEventListener - -Review the following guidelines before using an AsyncEventListener: - -- If you use an `AsyncEventListener` to implement a write-behind cache listener, your code should check for the possibility that an existing database connection may have been closed due to an earlier exception. For example, check for `Connection.isClosed()` in a catch block and re-create the connection as needed before performing further operations. -- Use a serial `AsyncEventQueue` if you need to preserve the order of region events within a thread when delivering events to your listener implementation. Use parallel queues when the order of events within a thread is not important, and when you require maximum throughput for processing events. In both cases, serial and parallel, the order of operations on a given key is preserved within the scope of the thread. -- You must install the `AsyncEventListener` implementation on a Geode member that hosts the region whose events you want to process. -- If you configure a parallel `AsyncEventQueue`, deploy the queue on each Geode member that hosts the region. -- You can install a listener on more than one member to provide high availability and guarantee delivery for events, in the event that a member with the active `AsyncEventListener` shuts down. At any given time only one member has an active listener for dispatching events. The listeners on other members remain on standby for redundancy. For best performance and most efficient use of memory, install only one standby listener (redundancy of at most one). -- Install no more than one standby listener (redundancy of at most one) for performance and memory reasons. -- To preserve pending events through member shutdowns, configure Geode to persist the internal queue of the `AsyncEventListener` to an available disk store. By default, any pending events that reside in the internal queue of an `AsyncEventListener` are lost if the active listener's member shuts down. -- To ensure high availability and reliable delivery of events, configure the event queue to be both persistent and redundant. - -## <a id="implementing_write_behind_cache_event_handling__section_FB3EB382E37945D9895E09B47A64D6B9" class="no-quick-link"></a>Implementing an AsyncEventListener - -To receive region events for processing, you create a class that implements the `AsyncEventListener` interface. The `processEvents` method in your listener receives a list of queued `AsyncEvent` objects in each batch. - -Each `AsyncEvent` object contains information about a region event, such as the name of the region where the event occurred, the type of region operation, and the affected key and value. - -The basic framework for implementing a write-behind event handler involves iterating through the batch of events and writing each event to a database. For example: - -``` pre -class MyAsyncEventListener implements AsyncEventListener { - - public boolean processEvents(List<AsyncEvent> events) { - - // Process each AsyncEvent - - for(AsyncEvent event: events) { - - // Write the event to a database - - } - } -} -``` - -## <a id="implementing_write_behind_cache_event_handling__section_AB80262CFB6D4867B52A5D6D880A5294" class="no-quick-link"></a>Processing AsyncEvents - -Use the [AsyncEventListener.processEvents](/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html) method to process AsyncEvents. This method is called asynchronously when events are queued to be processed. The size of the list reflects the number of batch events where batch size is defined in the AsyncEventQueueFactory. The `processEvents` method returns a boolean; true if the AsyncEvents are processed correctly, and false if any events fail processing. As long as `processEvents` returns false, Geode continues to re-try processing the events. - -You can use the `getDeserializedValue` method to obtain cache values for entries that have been updated or created. Since the `getDeserializedValue` method will return a null value for destroyed entries, you should use the `getKey` method to obtain references to cache objects that have been destroyed. Here's an example of processing AsyncEvents: - -``` pre -public boolean processEvents(@SuppressWarnings("rawtypes") List<AsyncEvent> list) - { - logger.log (Level.INFO, String.format("Size of List<GatewayEvent> = %s", list.size())); - List<JdbcBatch> newEntries = new ArrayList<JdbcBatch>(); - - List<JdbcBatch> updatedEntries = new ArrayList<JdbcBatch>(); - List<String> destroyedEntries = new ArrayList<String>(); - int possibleDuplicates = 0; - - for (@SuppressWarnings("rawtypes") AsyncEvent ge: list) - { - - if (ge.getPossibleDuplicate()) - possibleDuplicates++; - - if ( ge.getOperation().equals(Operation.UPDATE)) - { - updatedEntries.add((JdbcBatch) ge.getDeserializedValue()); - } - else if ( ge.getOperation().equals(Operation.CREATE)) - { - newEntries.add((JdbcBatch) ge.getDeserializedValue()); - } - else if ( ge.getOperation().equals(Operation.DESTROY)) - { - destroyedEntries.add(ge.getKey().toString()); - } - - } -``` - -## <a id="implementing_write_behind_cache_event_handling__section_9286E8C6B3C54089888E1680B4F43692" class="no-quick-link"></a>Configuring an AsyncEventListener - -To configure a write-behind cache listener, you first configure an asynchronous queue to dispatch the region events, and then create the queue with your listener implementation. You then assign the queue to a region in order to process that region's events. - -**Procedure** - -1. Configure a unique `AsyncEventQueue` with the name of your listener implementation. You can optionally configure the queue for parallel operation, persistence, batch size, and maximum memory size. See [WAN Configuration](../../reference/topics/elements_ref.html#topic_7B1CABCAD056499AA57AF3CFDBF8ABE3) for more information. - - **gfsh configuration** - - ``` pre - gfsh>create async-event-queue --id=sampleQueue --persistent --disk-store=exampleStore --listener=com.myCompany.MyAsyncEventListener --listener-param=url#jdbc:db2:SAMPLE,username#gfeadmin,password#admin1 - ``` - - The parameters for this command uses the following syntax: - - ``` pre - create async-event-queue --id=value --listener=value [--group=value] [--batch-size=value] - [--persistent(=value)?] [--disk-store=value] [--max-queue-memory=value] [--listener-param=value(,value)*] - ``` - - For more information, see [create async-event-queue](../../tools_modules/gfsh/command-pages/create.html#topic_ryz_pb1_dk). - - **cache.xml Configuration** - - ``` pre - <cache> - <async-event-queue id="sampleQueue" persistent="true" - disk-store-name="exampleStore" parallel="false"> - <async-event-listener> - <class-name>MyAsyncEventListener</class-name> - <parameter name="url"> - <string>jdbc:db2:SAMPLE</string> - </parameter> - <parameter name="username"> - <string>gfeadmin</string> - </parameter> - <parameter name="password"> - <string>admin1</string> - </parameter> - </async-event-listener> - </async-event-queue> - ... - </cache> - ``` - - **Java Configuration** - - ``` pre - Cache cache = new CacheFactory().create(); - AsyncEventQueueFactory factory = cache.createAsyncEventQueueFactory(); - factory.setPersistent(true); - factory.setDiskStoreName("exampleStore"); - factory.setParallel(false); - AsyncEventListener listener = new MyAsyncEventListener(); - AsyncEventQueue asyncQueue = factory.create("sampleQueue", listener); - ``` - -2. If you are using a parallel `AsyncEventQueue`, the gfsh example above requires no alteration, as gfsh applies to all members. If using cache.xml or the Java API to configure your `AsyncEventQueue`, repeat the above configuration in each Geode member that will host the region. Use the same ID and configuration settings for each queue configuration. - **Note:** - You can ensure other members use the sample configuration by using the cluster configuration service available in gfsh. See [Overview of the Cluster Configuration Service](../../configuring/cluster_config/gfsh_persist.html). - -3. On each Geode member that hosts the `AsyncEventQueue`, assign the queue to each region that you want to use with the `AsyncEventListener` implementation. - - **gfsh Configuration** - - ``` pre - gfsh>create region --name=Customer --async-event-queue-id=sampleQueue - ``` - - Note that you can specify multiple queues on the command line in a comma-delimited list. - - **cache.xml Configuration** - - ``` pre - <cache> - <region name="Customer"> - <region-attributes async-event-queue-ids="sampleQueue"> - </region-attributes> - </region> - ... - </cache> - ``` - - **Java Configuration** - - ``` pre - RegionFactory rf1 = cache.createRegionFactory(); - rf1.addAsyncEventQueue(sampleQueue); - Region customer = rf1.create("Customer"); - - // Assign the queue to multiple regions as needed - RegionFactory rf2 = cache.createRegionFactory(); - rf2.addAsyncEventQueue(sampleQueue); - Region order = rf2.create("Order"); - ``` - - Using the Java API, you can also add and remove queues to regions that have already been created: - - ``` pre - AttributesMutator mutator = order.getAttributesMutator(); - mutator.addAsyncEventQueueId("sampleQueue"); - ``` - - See the [Geode API documentation](/releases/latest/javadoc/org/apache/geode/cache/AttributesMutator.html) for more information. - -4. Optionally configure persistence and conflation for the queue. - **Note:** - You must configure your AsyncEventQueue to be persistent if you are using persistent data regions. Using a non-persistent queue with a persistent region is not supported. - -5. Optionally configure multiple dispatcher threads and the ordering policy for the queue using the instructions in [Configuring Dispatcher Threads and Order Policy for Event Distribution](configuring_gateway_concurrency_levels.html). - -The `AsyncEventListener` receives events from every region configured with the associated `AsyncEventQueue`.
