http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/extending_the_autoserializer.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/extending_the_autoserializer.html.md.erb b/geode-docs/developing/data_serialization/extending_the_autoserializer.html.md.erb new file mode 100644 index 0000000..35cbe66 --- /dev/null +++ b/geode-docs/developing/data_serialization/extending_the_autoserializer.html.md.erb @@ -0,0 +1,106 @@ +--- +title: Extending the ReflectionBasedAutoSerializer +--- + +You can extend the `ReflectionBasedAutoSerializer` to handle serialization in a customized manner. This section provides an overview of the available method-based customization options and an example of extending the serializer to support BigDecimal and BigInteger types. + +## <a id="concept_9E020566EE794A81A48A90BA798EC279__section_378C0C68A1B342DD9D754DDBDC4874B3" class="no-quick-link"></a>Reasons to Extend the ReflectionBasedAutoSerializer + +One of the main use cases for extending the `ReflectionBasedAutoSerializer` is that you want it to handle an object that would currently need to be handled by standard Java serialization. There are several issues with having to use standard Java serialization that can be addressed by extending the PDX `ReflectionBasedAutoSerializer`. + +- Each time we transition from a Geode serialized object to an object that will be Java I/O serialized, extra data must get serialized. This can cause a great deal of serialization overhead. This is why it is worth extending the `ReflectionBasedAutoSerializer` to handle any classes that normally would have to be Java I/O serialized. +- Expanding the number of classes that can use the `ReflectionBasedAutoSerializer` is beneficial when you encounter object graphs. After we use Java I/O serialization on an object, any objects under that object in the object graph will also have to be Java I/O serialized. This includes objects that normally would have been serialized using PDX or `DataSerializable`. +- If standard Java I/O serialization is done on an object and you have enabled check-portability, then an exception will be thrown. Even if you are not concerned with the object's portability, you can use this flag to find out what classes would use standard Java serialization (by getting an exception on them) and then enhancing your auto serializer to handle them. + +## <a id="concept_9E020566EE794A81A48A90BA798EC279__section_A739691C60FB4EB291289AADCD66C675" class="no-quick-link"></a>Overriding ReflectionBasedAutoSerializer Behavior + +You can customize the specific behaviors in `ReflectionBasedAutoSerializer` by overriding the following methods: + +- **`isClassAutoSerialized`** customizes which classes to autoserialize. +- **`isFieldIncluded`** specifies which fields of a class to autoserialize. +- **`getFieldName`** defines the specific field names that will be generated during autoserialization. +- **`isIdentifyField`** controls which field is marked as the identity field. Identity fields are used when a PdxInstance computes its hash code to determine whether it is equal to another object. +- **`getFieldType`** determines the field type that will be used when autoserializing the given field. +- **`transformFieldValue`** controls whether specific field values of a PDX object can be transformed during serialization. +- **`writeTransform`** controls what field value is written during auto serialization. +- **`readTransform`** controls what field value is read during auto deserialization. + +These methods are only called the first time the `ReflectionBasedAutoSerializer` sees a new class. The results will be remembered and used the next time the same class is seen. + +For details on these methods and their default behaviors, see the JavaDocs on [ReflectionBasedAutoSerializer](/releases/latest/javadoc/org/apache/geode/pdx/ReflectionBasedAutoSerializer.html) for details. + +## <a id="concept_9E020566EE794A81A48A90BA798EC279__section_7C4CC39FD82A48A9B5F8376522078192" class="no-quick-link"></a>Example of Optimizing Autoserialization of BigInteger and BigDecimal Types + +This section provides an example of extending the `ReflectionBasedAutoSerializer` to optimize the automatic serialization of BigInteger and BigDecimal types. + +The following code sample illustrates a subclass of the `ReflectionBasedAutoSerializer` that optimizes BigInteger and BigDecimal autoserialization: + +``` pre +public static class BigAutoSerializer extends ReflectionBasedAutoSerializer { + public BigAutoSerializer(Boolean checkPortability, string⦠patterns) { + super(checkPortability, patterns); +} + +@Override +public FieldType get FieldType(Field f, Class<?> clazz) { + if (f.getType().equals(BigInteger.class)) { + return FieldType.BYTE_ARRAY; + } else if (f.getType().equals(BigDecimal.class)) { + return FieldType.STRING; + } else { + return super.getFieldType(f, clazz); + } + } +@Override + public boolean transformFieldValue(Field f, Class<?> clazz) { + if (f.getType().equals(BigInteger.class)) { + return true; + } else if (f.getType().equals(BigDecimal.class)) { + return true; + } else { + return super.transformFieldValue(f, clazz); + } + } + +@Override + public Object writeTransform(Field f, Class<?> clazz, Object originalValue) { + if (f.getType().equals(BigInteger.class)) { + byte[] result = null; + if (originalValue != null) { + BigInteger bi = (BigInteger)originalValue; + result = bi.toByteArray(); + } + return result; + } else if (f.getType().equals(BigDecimal.class)) { + Object result = null; + if (originalValue != null) { + BigDecimal bd = (BigDecimal)originalValue; + result = bd.toString(); + } + return result; + } else { + return super.writeTransform(f, clazz, originalValue); + } + } + +@Override + public Object readTransform(Field f, Class<?> clazz, Object serializedValue) { + if (f.getType().equals(BigInteger.class)) { + BigInteger result = null; + if (serializedValue != null) { + result = new BigInteger((byte[])serializedValue); + } + return result; + } else if (f.getType().equals(BigDecimal.class)) { + BigDecimal result = null; + if (serializedValue != null) { + result = new BigDecimal((String)serializedValue); + } + return result; + } else { + return super.readTransform(f, clazz, serializedValue); + } + } + + } +```
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/gemfire_data_serialization.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/gemfire_data_serialization.html.md.erb b/geode-docs/developing/data_serialization/gemfire_data_serialization.html.md.erb new file mode 100644 index 0000000..58aeb7e --- /dev/null +++ b/geode-docs/developing/data_serialization/gemfire_data_serialization.html.md.erb @@ -0,0 +1,35 @@ +--- +title: Geode Data Serialization (DataSerializable and DataSerializer) +--- + +Geode's `DataSerializable` interface gives you quick serialization of your objects. + +## <a id="gemfire_data_serialization__section_0C84D6BF5E9748CB865E6BB944A077DE" class="no-quick-link"></a>Data Serialization with the DataSerializable Interface + +Geode's `DataSerializable` interface gives you faster and more compact data serialization than the standard Java serialization or Geode PDX serialization. However, while Geode `DataSerializable` interface is generally more performant than Geode's `PdxSerializable`, it requires full deserialization on the server and then reserialization to send the data back to the client. + +You can further speed serialization by registering the instantiator for your `DataSerializable` class through `Instantiator`, eliminating the need for reflection to find the right serializer. You can provide your own serialization through the API. + +The recommended way to register your custom `Instantiator` is by specifying it in the `serialization-registration` element of cache.xml. + +For more information, see the online Java documentation for `DataSerializable` and `DataSerializer`. + +**Example cache.xml:** + +The following provides an example of how to register an instantiator using cache.xml. + +``` pre +<serialization-registration> +<instantiator id="30"> + <class-name>com.package.MyClass</class-name> +</instantiator> +</serialization-registration> +``` + +In addition to speeding standard object serialization, you can use the `DataSerializable` interface to serialize any custom objects you store in the cache. + +## <a id="gemfire_data_serialization__section_B21408E7090C41B08BF300146F87648B" class="no-quick-link"></a>Serializing Your Domain Object with DataSerializer + +You can also use `DataSerializer` to serialize domain objects. It serializes data in the same way as `DataSerializable` but allows you to serialize classes without modifying the domain class code. + +See the JavaDocs on [DataSerializable](/releases/latest/javadoc/org/apache/geode/DataSerializable.html) and [DataSerializer](/releases/latest/javadoc/org/apache/geode/DataSerializer.html) for more information. http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/gemfire_pdx_serialization.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/gemfire_pdx_serialization.html.md.erb b/geode-docs/developing/data_serialization/gemfire_pdx_serialization.html.md.erb new file mode 100644 index 0000000..1053618 --- /dev/null +++ b/geode-docs/developing/data_serialization/gemfire_pdx_serialization.html.md.erb @@ -0,0 +1,47 @@ +--- +title: Geode PDX Serialization +--- + +Geode's Portable Data eXchange (PDX) is a cross-language data format that can reduce the cost of distributing and serializing your objects. PDX stores data in named fields that you can access individually, to avoid the cost of deserializing the entire data object. PDX also allows you to mix versions of objects where you have added or removed fields. + +- **[Geode PDX Serialization Features](../../developing/data_serialization/PDX_Serialization_Features.html)** + + Geode PDX serialization offers several advantages in terms of functionality. + +- **[High Level Steps for Using PDX Serialization](../../developing/data_serialization/use_pdx_high_level_steps.html)** + + To use PDX serialization, you can configure and use Geode's reflection-based autoserializer, or you can program the serialization of your objects by using the PDX interfaces and classes. + +- **[Using Automatic Reflection-Based PDX Serialization](../../developing/data_serialization/auto_serialization.html)** + + You can configure your cache to automatically serialize and deserialize domain objects without having to add any extra code to them. + +- **[Serializing Your Domain Object with a PdxSerializer](../../developing/data_serialization/use_pdx_serializer.html)** + + For a domain object that you cannot or do not want to modify, use the `PdxSerializer` class to serialize and deserialize the object's fields. You use one `PdxSerializer` implementation for the entire cache, programming it for all of the domain objects that you handle in this way. + +- **[Implementing PdxSerializable in Your Domain Object](../../developing/data_serialization/use_pdx_serializable.html)** + + For a domain object with source that you can modify, implement the `PdxSerializable` interface in the object and use its methods to serialize and deserialize the object's fields. + +- **[Programming Your Application to Use PdxInstances](../../developing/data_serialization/program_application_for_pdx.html)** + + A `PdxInstance` is a light-weight wrapper around PDX serialized bytes. It provides applications with run-time access to fields of a PDX serialized object. + +- **[Adding JSON Documents to the Geode Cache](../../developing/data_serialization/jsonformatter_pdxinstances.html)** + + The `JSONFormatter` API allows you to put JSON formatted documents into regions and retrieve them later by storing the documents internally as PdxInstances. + +- **[Using PdxInstanceFactory to Create PdxInstances](../../developing/data_serialization/using_PdxInstanceFactory.html)** + + You can use the `PdxInstanceFactory` interface to create a `PdxInstance` from raw data when the domain class is not available on the server. + +- **[Persisting PDX Metadata to Disk](../../developing/data_serialization/persist_pdx_metadata_to_disk.html)** + + Geode allows you to persist PDX metadata to disk and specify the disk store to use. + +- **[Using PDX Objects as Region Entry Keys](../../developing/data_serialization/using_pdx_region_entry_keys.html)** + + Using PDX objects as region entry keys is highly discouraged. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/java_serialization.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/java_serialization.html.md.erb b/geode-docs/developing/data_serialization/java_serialization.html.md.erb new file mode 100644 index 0000000..14b9ea3 --- /dev/null +++ b/geode-docs/developing/data_serialization/java_serialization.html.md.erb @@ -0,0 +1,12 @@ +--- +title: Standard Java Serialization +--- + +You can use standard Java serialization for data you only distribute between Java applications. If you distribute your data between non-Java clients and Java servers, you need to do additional programming to get the data between the various class formats. + +<a id="java_serialization__section_AD2524E4E3C145D4A2CEB379DD8E9389"></a> +Standard Java types are serializable by definition. For your domain classes, implement `java.io.Serializable`, then make sure to mark your transient and static variables as needed for your objects. For information, see the online documentation for `java.io.Serializable` for your Java version. + +Mixing `DataSerializable` with `Serializable` or `PdxSerializable` use on the same data can result in increased memory use and lower throughput than using just `Serializable` on the entire data, especially if the `Serializable` entries are in collections. The bigger the data collection, the lower the throughput as the metadata for the collection entries is not shared when using `DataSerializable`. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/jsonformatter_pdxinstances.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/jsonformatter_pdxinstances.html.md.erb b/geode-docs/developing/data_serialization/jsonformatter_pdxinstances.html.md.erb new file mode 100644 index 0000000..59ceb3c --- /dev/null +++ b/geode-docs/developing/data_serialization/jsonformatter_pdxinstances.html.md.erb @@ -0,0 +1,29 @@ +--- +title: Adding JSON Documents to the Geode Cache +--- + +The `JSONFormatter` API allows you to put JSON formatted documents into regions and retrieve them later by storing the documents internally as PdxInstances. + +Geode now supports the use of JSON formatted documents natively. When you add a JSON document to a Geode cache, you call the JSONFormatter APIs to transform them into the PDX format (as a `PdxInstance`), which enables Geode to understand the JSON document at a field level. + +In terms of querying and indexing, because the documents are stored internally as PDX, applications can index on any field contained inside the JSON document including any nested field (within JSON objects or JSON arrays.) Any queries run on these stored documents will return PdxInstances as results. To update a JSON document stored in Geode , you can execute a function on the PdxInstance. + +You can then use the `JSONFormatter` to convert the PdxInstance results back into the JSON document. + +`JSONFormatter` uses a streaming parser ([Jackson](http://wiki.fasterxml.com/JacksonHome), JSON processor) to turn JSON documents into the optimized PDX format. To use the JSONFormatter, make sure that `$GEMFIRE/lib/server-dependencies.jar` is available in your application's CLASSPATH. + +The `JSONFormatter` class has four static methods that are used to convert JSON document into PdxInstances and then to convert those PdxInstances back into JSON document. + +You need to call the following methods before putting any JSON document into the Geode region: + +- `fromJSON`. Creates a PdxInstance from a JSON byte array. Returns the PdxInstance. +- `fromJSON`. Creates a PdxInstance from a JSON string. Returns the PdxInstance. + +After putting the JSON document into a region as a PdxInstance, you can execute standard Geode queries and create indexes on the JSON document in the same manner you would query or index any other Geode PdxInstance. + +After executing a Geode query or calling `region.get`, you can use the following methods to convert a PdxInstance back into the JSON format: + +- `toJSON`. Reads a PdxInstance and returns a JSON string. +- `toJSONByteArray`. Reads a PdxInstance and returns a JSON byte array. + +For more information on using the JSONFormatter, see the Java API documentation for `org.apache.geode.pdx.JSONFormatter`. http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/persist_pdx_metadata_to_disk.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/persist_pdx_metadata_to_disk.html.md.erb b/geode-docs/developing/data_serialization/persist_pdx_metadata_to_disk.html.md.erb new file mode 100644 index 0000000..37b11fc --- /dev/null +++ b/geode-docs/developing/data_serialization/persist_pdx_metadata_to_disk.html.md.erb @@ -0,0 +1,36 @@ +--- +title: Persisting PDX Metadata to Disk +--- + +Geode allows you to persist PDX metadata to disk and specify the disk store to use. + +<a id="persist_pdx_metadata_to_disk__section_7F357A8E56B54BFB9A5778C0F89E034E"></a> +**Prerequisites** + +- Understand generally how to configure the Geode cache. See [Basic Configuration and Programming](../../basic_config/book_intro.html). +- Understand how Geode disk stores work. See [Disk Storage](../../managing/disk_storage/chapter_overview.html). + +**Procedure** + +1. Set the `<pdx>` attribute `persistent` to true in your cache configuration. This is required for caches that use PDX with persistent regions and with regions that use a gateway sender to distribute events across a WAN.. Otherwise, it is optional. +2. (Optional) If you want to use a disk store that is not the Geode default disk store, set the `<pdx>` attribute `disk-store-name` to the name of your non-default disk store. + **Note:** + If you are using PDX serialized objects as region entry keys and you are using persistent regions, then you must configure your PDX disk store to be a different one than the disk store used by the persistent regions. + +3. (Optional) If you later want to rename the PDX types that are persisted to disk, you can do so on your offline disk-stores by executing the `pdx rename` command. See [pdx rename](../../tools_modules/gfsh/command-pages/pdx.html). + +**Example cache.xml:** + +This example `cache.xml` enables PDX persistence and sets a non-default disk store in a server cache configuration: + +``` pre + <pdx read-serialized="true" + persistent="true" disk-store-name="SerializationDiskStore"> + <pdx-serializer> + <class-name>pdxSerialization.defaultSerializer</class-name> + </pdx-serializer> + </pdx> + <region ... +``` + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/program_application_for_pdx.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/program_application_for_pdx.html.md.erb b/geode-docs/developing/data_serialization/program_application_for_pdx.html.md.erb new file mode 100644 index 0000000..9bfbe12 --- /dev/null +++ b/geode-docs/developing/data_serialization/program_application_for_pdx.html.md.erb @@ -0,0 +1,90 @@ +--- +title: Programming Your Application to Use PdxInstances +--- + +A `PdxInstance` is a light-weight wrapper around PDX serialized bytes. It provides applications with run-time access to fields of a PDX serialized object. + +<a id="program_application_for_pdx__section_7F357A8E56B54BFB9A5778C0F89E034E"></a> +You can configure your cache to return a `PdxInstance` when a PDX serialized object is deserialized instead of deserializing the object to a domain class. You can then program your application code that reads your entries to handle `PdxInstance`s fetched from the cache. + +**Note:** +This applies only to entry retrieval that you explicitly code using methods like `EntryEvent.getNewValue` and `Region.get`, as you do inside functions or in cache listener code. This does not apply to querying because the query engine retrieves the entries and handles object access for you. + +If you configure your cache to allow PDX serialized reads, a fetch from the cache returns the data in the form it is found. If the object is not serialized, the fetch returns the domain object. If the object is serialized, the fetch returns the `PdxInstance` for the object. + +**Note:** +If you are using `PdxInstance`s, you cannot use delta propagation to apply changes to PDX serialized objects. + +For example, in client/server applications that are programmed and configured to handle all data activity from the client, PDX serialized reads done on the server side will always return a `PdxInstance`. This is because all of data is serialized for transfer from the client, and you are not performing any server-side activities that would deserialize the objects in the server cache. + +In mixed situations, such as where a server cache is populated from client operations and also from data loads done on the server side, fetches done on the server can return a mix of `PdxInstance`s and domain objects. + +When fetching data in a cache with PDX serialized reads enabled, the safest approach is to code to handle both types, receiving an `Object` from the fetch operation, checking the type and casting as appropriate. However, if you know that the class is not available in the JVM, then you can avoid performing the type check. + +`PdxInstance` overrides any custom implementation you might have coded for your object's `equals` and `hashcode` methods. Make sure you have marked at least one identity field when writing PDX serialized objects. If you do not set as least one identity field, then the PdxInstance`equals` and `hashCode` methods will use all PDX fields to compare objects and consequently, will not perform as well. + +<a id="program_application_for_pdx__section_03F8918B4CAD49CCA1823FAAE25D53A8"></a> + +**Prerequisites** + +- Understand generally how to configure the Geode cache. See [Basic Configuration and Programming](../../basic_config/book_intro.html#basic_config_management). + +<a id="program_application_for_pdx__section_B3C7C7629DFD4483B32B27F84D64DFCF"></a> + +**Procedure** + +In your application where you fetch data from the cache, provide the following configuration and code as appropriate: + +1. In the cache.xml file of the member where entry fetches are run, set the `<pdx>` `read-serialized` attribute to true. + Data is not necessarily accessed on the member that you have coded for it. For example, if a client application runs a function on a server, the actual data access is done on the server, so you set `read-serialized` to true on the server. + + For example: + + ``` pre + // Cache configuration setting PDX read behavior + <cache> + <pdx read-serialized="true" /> + ... + </cache> + + ``` + +2. Write the application code that fetches data from the cache to handle a `PdxInstance`. If you are sure you will only retrieve `PdxInstance`s from the cache, you can code only for that. In many cases, a `PdxInstance` or a domain object may be returned from your cache entry retrieval operation, so you should check the object type and handle each possible type. + + For example: + + ``` pre + // put/get code with serialized read behavior + // put is done as normal + myRegion.put(myKey, myPdxSerializableObject); + + // get checks Object type and handles each appropriately + Object myObject = myRegion.get(myKey); + if (myObject instanceof PdxInstance) { + // get returned PdxInstance instead of domain object + PdxInstance myPdxInstance = (PdxInstance)myObject; + + // PdxInstance.getField deserializes the field, but not the object + String fieldValue = myPdxInstance.getField("stringFieldName"); + + // Update a field and put it back into the cache + // without deserializing the entire object + WritablePdxInstance myWritablePdxI = myPdxInstance.createWriter(); + myWritablePdxI.setField("fieldName", fieldValue); + region.put(key, myWritablePdxI); + + // Deserialize the entire object if needed, from the PdxInstance + DomainClass myPdxObject = (DomainClass)myPdxInstance.getObject(); + } + else if (myObject instanceof DomainClass) { + // get returned instance of domain object + // code to handle domain object instance + ... + } + ... + ``` + + **Note:** + Due to a limitation with PDX, if your PDX-enabled cache contains TreeSet domain objects, you should implement a Comparator that can handle both your domain objects and PdxInstance objects. You will also need to make the domain classes available on the server. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/use_pdx_high_level_steps.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/use_pdx_high_level_steps.html.md.erb b/geode-docs/developing/data_serialization/use_pdx_high_level_steps.html.md.erb new file mode 100644 index 0000000..f47e81f --- /dev/null +++ b/geode-docs/developing/data_serialization/use_pdx_high_level_steps.html.md.erb @@ -0,0 +1,32 @@ +--- +title: High Level Steps for Using PDX Serialization +--- + +To use PDX serialization, you can configure and use Geode's reflection-based autoserializer, or you can program the serialization of your objects by using the PDX interfaces and classes. + +<a id="concept_A7C8890826394B4293C036DD739835BD__section_7F357A8E56B54BFB9A5778C0F89E034E"></a> +Optionally, program your application code to deserialize individual fields out of PDX representations of your serialized objects. You may also need to persist your PDX metadata to disk for recovery on startup. + +**Procedure** + +1. Use one of these serialization options for each object type that you want to serialize using PDX serialization: + - [Using Automatic Reflection-Based PDX Serialization](auto_serialization.html) + - [Serializing Your Domain Object with a PdxSerializer](use_pdx_serializer.html) + - [Implementing PdxSerializable in Your Domain Object](use_pdx_serializable.html) + +2. To ensure that your servers do not need to load the application classes, set the `pdx` `read-serialized` attribute to true. In gfsh, execute the following command before starting up your servers: + + ``` pre + gfsh>configure pdx --read-serialized=true + ``` + + By using gfsh, this configuration can propagated across the cluster through the [Cluster Configuration Service](../../configuring/cluster_config/gfsh_persist.html). Alternately, you would need to configure `pdx read-serialized` in each server's `cache.xml` file. + +3. If you are storing any Geode data on disk, then you must configure PDX serialization to use persistence. See [Persisting PDX Metadata to Disk](persist_pdx_metadata_to_disk.html) for more information. +4. (Optional) Wherever you run explicit application code to retrieve and manage your cached entries, you may want to manage your data objects without using full deserialization. To do this, see [Programming Your Application to Use PdxInstances](program_application_for_pdx.html). + +## PDX and Multi-Site (WAN) Deployments + +For multisite (WAN) installations only-- if you will use PDX serialization in any of your WAN-enabled regions, for each distributed system, you must choose a unique integer between 0 (zero) and 255 and set the `distributed-system-id` in every member's `gemfire.properties` file. See [Configuring a Multi-site (WAN) System](../../topologies_and_comm/multi_site_configuration/setting_up_a_multisite_system.html). + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/use_pdx_serializable.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/use_pdx_serializable.html.md.erb b/geode-docs/developing/data_serialization/use_pdx_serializable.html.md.erb new file mode 100644 index 0000000..948112b --- /dev/null +++ b/geode-docs/developing/data_serialization/use_pdx_serializable.html.md.erb @@ -0,0 +1,98 @@ +--- +title: Implementing PdxSerializable in Your Domain Object +--- + +For a domain object with source that you can modify, implement the `PdxSerializable` interface in the object and use its methods to serialize and deserialize the object's fields. + +<a id="use_pdx_serializable__section_7F357A8E56B54BFB9A5778C0F89E034E"></a> +**Procedure** + +1. In your domain class, implement `PdxSerializable`, importing the required `org.apache.geode.pdx` classes. + + For example: + + ``` pre + import org.apache.geode.pdx.PdxReader; + import org.apache.geode.pdx.PdxSerializable; + import org.apache.geode.pdx.PdxWriter; + + public class PortfolioPdx implements PdxSerializable { + ... + ``` + +2. If your domain class does not have a zero-arg constructor, create one for it. + + For example: + + ``` pre + public PortfolioPdx(){} + ``` + +3. Program `PdxSerializable.toData.` + 1. Write each standard Java data field of your domain class using the `PdxWriter` write methods. Geode automatically provides `PdxWriter` to the `toData` method for `PdxSerializable` objects. + 2. Call the `PdxWriter` `markIdentifyField` method for each field you want to have Geode use to identify your object. Put this after the field's write method. Geode uses this information to compare objects for operations like distinct queries. If you do not set as least one identity field, then the `equals` and `hashCode` methods will use all PDX fields to compare objects and consequently, will not perform as well. It is important that the fields used by your `equals` and `hashCode` implementations are the same fields that you mark as identity fields. + 3. For a particular version of your class, you need to consistently write the same named field each time. The field names or number of fields must not change from one instance to another for the same class version. + 4. For best performance, do fixed width fields first and then variable length fields. + + Example `toData` code: + + ``` pre + // PortfolioPdx fields + private int id; + private String pkid; + private Map<String, PositionPdx> positions; + private String type; + private String status; + private String[] names; + private byte[] newVal; + private Date creationDate; + ... + + public void toData(PdxWriter writer) + { + writer.writeInt("id", id) + // The markIdentifyField call for a field must + // come after the field's write method + .markIdentityField("id") + .writeDate("creationDate", creationDate) //fixed length field + .writeString("pkid", pkid) + .writeObject("positions", positions) + .writeString("type", type) + .writeString("status", status) + .writeStringArray("names", names) + .writeByteArray("newVal", newVal) + } + ``` + +4. Program `PdxSerializable.fromData` to read your data fields from the serialized form into the object's fields using the `PdxReader` read methods. + + Provide the same names that you did in `toData` and call the read operations in the same order as you called the write operations in your `toData` implementation. + + Geode automatically provides `PdxReader` to the `fromData` method for `PdxSerializable` objects. + + Example `fromData` code: + + ``` pre + + public void fromData(PdxReader reader) + { + id = reader.readInt("id"); + creationDate = reader.readDate("creationDate"); + pkid = reader.readString("pkid"); + position1 = (PositionPdx)reader.readObject("position1"); + position2 = (PositionPdx)reader.readObject("position2"); + positions = (Map<String, PositionPdx>)reader.readObject("positions"); + type = reader.readString("type"); + status = reader.readString("status"); + names = reader.readStringArray("names"); + newVal = reader.readByteArray("newVal"); + arrayNull = reader.readByteArray("arrayNull"); + arrayZeroSize = reader.readByteArray("arrayZeroSize"); + } + ``` + +**What to do next** + +- As needed, configure and program your Geode applications to use `PdxInstance` for selective object deserialization. See [Programming Your Application to Use PdxInstances](program_application_for_pdx.html). + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/use_pdx_serializer.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/use_pdx_serializer.html.md.erb b/geode-docs/developing/data_serialization/use_pdx_serializer.html.md.erb new file mode 100644 index 0000000..7ef401e --- /dev/null +++ b/geode-docs/developing/data_serialization/use_pdx_serializer.html.md.erb @@ -0,0 +1,128 @@ +--- +title: Serializing Your Domain Object with a PdxSerializer +--- + +For a domain object that you cannot or do not want to modify, use the `PdxSerializer` class to serialize and deserialize the object's fields. You use one `PdxSerializer` implementation for the entire cache, programming it for all of the domain objects that you handle in this way. + +With `PdxSerializer`, you leave your domain object as-is and handle the serialization and deserialization in the separate serializer. You register the serializer in your cache PDX configuration. Program the serializer to handle all of the domain objects you need. + +If you write your own `PdxSerializer` and you also use the `ReflectionBasedAutoSerializer`, then the `PdxSerializer` needs to own the `ReflectionBasedAutoSerializer` and delegate to it. A Cache can only have a single `PdxSerializer` instance. + +**Note:** +The `PdxSerializer` `toData` and `fromData` methods differ from those for `PdxSerializable`. They have different parameters and results. + +**Procedure** + +1. In the domain classes that you wish to PDX serialize, make sure each class has a zero-arg constructor. For example: + + ``` pre + public PortfolioPdx(){} + ``` + +2. If you have not already implemented `PdxSerializer` for some other domain object, perform these steps: + 1. Create a new class as your cache-wide serializer and make it implement `PdxSerializer`. If you want to declare your new class in the `cache.xml` file, have it also implement `Declarable`. + + Example: + + ``` pre + import org.apache.geode.cache.Declarable; + import org.apache.geode.pdx.PdxReader; + import org.apache.geode.pdx.PdxSerializer; + import org.apache.geode.pdx.PdxWriter; + + public class ExamplePdxSerializer implements PdxSerializer, Declarable { + ... + ``` + + 2. In your cache pdx configuration, register the serializer class in the cache's `<pdx>` `<pdx-serializer>` `<class-name>` attribute. + + Example: + + ``` pre + // Configuration setting PDX serializer for the cache + <cache> + <pdx> + <pdx-serializer> + <class-name>com.company.ExamplePdxSerializer</class-name> + </pdx-serializer> + </pdx> + ... + </cache> + ``` + + Or use the `CacheFactory.setPdxSerializer ` API. + + ``` pre + Cache c = new CacheFactory + .setPdxSerializer(new ExamplePdxSerializer()) + .create(); + ``` + + **Note:** + You cannot specify a custom `pdx-serializer` class using gfsh, however the `configure pdx` command automatically configures the <span class="keyword apiname">org.apache.geode.pdx.ReflectionBasedAutoSerializer</span> class. See [configure pdx](../../tools_modules/gfsh/command-pages/configure.html#topic_jdkdiqbgphqh). + +3. Program `PdxSerializer.toData` to recognize, cast, and handle your domain object: + + 1. Write each standard Java data field of your domain class using the `PdxWriter` write methods. + 2. Call the `PdxWriter` `markIdentityField` method for each field you want to have Geode use to identify your object. Put this after the field's write method. Geode uses this information to compare objects for operations like distinct queries. If you do not set as least one identity field, then the `equals` and `hashCode` methods will use all PDX fields to compare objects and consequently, will not perform as well. It is important that the fields used by your `equals` and `hashCode` implementations are the same fields that you mark as identity fields. + 3. For a particular version of your class, you need to consistently write the same named field each time. The field names or number of fields must not change from one instance to another for the same class version. + 4. For best performance, do fixed width fields first and then variable length fields. + 5. If desired, you can check the portability of the object before serializing it by adding the `checkPortability` parameter when using the` PdxWriter` `writeObject`, `writeObjectArray`, and `writeField` methods. + + Example `toData` code: + + ``` pre + public boolean toData(Object o, PdxWriter writer) + { + if(!(o instanceof PortfolioPdx)) { + return false; + } + + PortfolioPdx instance = (PortfolioPdx) o; + writer.writeInt("id", instance.id) + //identity field + .markIdentityField("id") + .writeDate("creationDate", instance.creationDate) + .writeString("pkid", instance.pkid) + .writeObject("positions", instance.positions) + .writeString("type", instance.type) + .writeString("status", instance.status) + .writeStringArray("names", instance.names) + .writeByteArray("newVal", instance.newVal) + + return true; + } + ``` + + 1. Program `PdxSerializer.fromData` to create an instance of your class, read your data fields from the serialized form into the object's fields using the `PdxReader` read methods, and return the created object. + + Provide the same names that you did in `toData` and call the read operations in the same order as you called the write operations in your `toData` implementation. + + Geode provides the domain class type and `PdxReader` to the `fromData` method. + + Example `fromData` code: + + ``` pre + public Object fromData(Class<?> clazz, PdxReader reader) + { + if(!clazz.equals(PortfolioPdx.class)) { + return null; + } + + PortfolioPdx instance = new PortfolioPdx(); + instance.id = reader.readInt("id"); + instance.creationDate = reader.readDate("creationDate"); + instance.pkid = reader.readString("pkid"); + instance.positions = (Map<String, PositionPdx>)reader.readObject("positions"); + instance.type = reader.readString("type"); + instance.status = reader.readString("status"); + instance.names = reader.readStringArray("names"); + instance.newVal = reader.readByteArray("newVal"); + + return instance; + } + ``` + +4. If desired, you can also enable extra validation in your use of `PdxWriter`. You can set this by setting the system property `gemfire.validatePdxWriters` to **true**. Note that you should only set this option if you are debugging new code as this option can decrease system performance. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/using_PdxInstanceFactory.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/using_PdxInstanceFactory.html.md.erb b/geode-docs/developing/data_serialization/using_PdxInstanceFactory.html.md.erb new file mode 100644 index 0000000..7b9501d --- /dev/null +++ b/geode-docs/developing/data_serialization/using_PdxInstanceFactory.html.md.erb @@ -0,0 +1,34 @@ +--- +title: Using PdxInstanceFactory to Create PdxInstances +--- + +You can use the `PdxInstanceFactory` interface to create a `PdxInstance` from raw data when the domain class is not available on the server. + +This can be particularly useful when you need an instance of a domain class for plug in code such as a function or a loader. If you have the raw data for the domain object (the class name and each field's type and data), then you can explicitly create a `PdxInstance`. The `PdxInstanceFactory` is very similar to the `PdxWriter` except that after writing each field, you need to call the create method which returns the created PdxInstance. + +To create a factory call `RegionService.createPdxInstanceFactory`. A factory can only create a single instance. To create multiple instances create multiple factories or use `PdxInstance.createWriter()` to create subsequent instances. Using `PdxInstance.createWriter()` is usually faster. + +When you create a PdxInstance, set as least one identity field using the `markIndentityField` method. If you do not mark an identity field, the PdxInstance`equals` and `hashCode` methods will use all PDX fields to compare objects and consequently, will not perform as well. It is important that the fields used by your `equals` and `hashCode` implementations are the same fields that you mark as identity fields. + +The following is a code example of using `PdxInstanceFactory`: + +``` pre +PdxInstance pi = cache.createPdxInstanceFactory("com.company.DomainObject") + .writeInt("id", 37) + .markIdentityField("id") + .writeString("name", "Mike Smith") + .writeObject("favoriteDay", cache.createPdxEnum("com.company.Day", "FRIDAY", 5)) + .create(); +``` + +For more information, see `PdxInstanceFactory` in the Java API documentation. + +## <a id="concept_FFECBE8249D848E9A2CF7FD02514EC68__section_F4EC56197730427084FBF040820A6149" class="no-quick-link"></a>Enum Objects as PdxInstances + +You can now work with enum objects as PdxInstances. When you fetch an enum object from the cache, you can now deserialize it as a `PdxInstance`. To check whether a `PdxInstance` is an enum, use the `PdxInstance.isEnum` method. An enum `PdxInstance` will have one field named "name" whose value is a String that corresponds to the enum constant name. + +An enum `PdxInstance` is not writable; if you call `createWriter` it will throw an exception. + +The `RegionService` has a method that allows you to create a `PdxInstance` that represents an enum. See `RegionService.createPdxEnum` in the Java API documentation. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/data_serialization/using_pdx_region_entry_keys.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/data_serialization/using_pdx_region_entry_keys.html.md.erb b/geode-docs/developing/data_serialization/using_pdx_region_entry_keys.html.md.erb new file mode 100644 index 0000000..91c91c4 --- /dev/null +++ b/geode-docs/developing/data_serialization/using_pdx_region_entry_keys.html.md.erb @@ -0,0 +1,14 @@ +--- +title: Using PDX Objects as Region Entry Keys +--- + +Using PDX objects as region entry keys is highly discouraged. + +<a id="concept_E5B95958B8D04B2285CB5E4DC6FABC78__section_900BD620D716407AA78BE0E38C2C396D"></a> +The best practice for creating region entry keys is to use a simple key; for example, use a String or Integer. If the key must be a domain class, then you should use a non-PDX-serialized class. + +If you must use PDX serialized objects as region entry keys, ensure that you do not set `read-serialized` to `true`. This configuration setting will cause problems in partitioned regions because partitioned regions require the hash code of the key to be the same on all JVMs in the distributed system. When the key is a `PdxInstance` object, its hash code will likely not be the same as the hash code of the domain object. + +If you are using PDX serialized objects as region entry keys and you are using persistent regions, then you must configure your PDX disk store to be a different one than the disk store used by the persistent regions. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/delta_propagation/chapter_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/delta_propagation/chapter_overview.html.md.erb b/geode-docs/developing/delta_propagation/chapter_overview.html.md.erb new file mode 100644 index 0000000..1beb847 --- /dev/null +++ b/geode-docs/developing/delta_propagation/chapter_overview.html.md.erb @@ -0,0 +1,31 @@ +--- +title: Delta Propagation +--- + +Delta propagation allows you to reduce the amount of data you send over the network by including only changes to objects rather than the entire object. + +- **[How Delta Propagation Works](how_delta_propagation_works.html)** + + Delta propagation reduces the amount of data you send over the network. You do this by only sending the change, or delta, information about an object, instead of sending the entire changed object. If you do not use cloning when applying the deltas, you can also expect to generate less garbage in your receiving JVMs. + +- **[When to Avoid Delta Propagation](when_to_use_delta_prop.html)** + + Generally, the larger your objects and the smaller the deltas, the greater the benefits of using delta propagation. Partitioned regions with higher redundancy levels generally benefit more from delta propagation. However, in some application scenarios, delta propagation does not show any significant benefits. On occasion it results in performance degradation. + +- **[Delta Propagation Properties](delta_propagation_properties.html)** + + This topic describes the properties that can be used to configure delta propagation. + +- **[Implementing Delta Propagation](implementing_delta_propagation.html)** + + By default, delta propagation is enabled in your distributed system. When enabled, delta propagation is used for objects that implement `org.apache.geode.Delta`. You program the methods to store and extract delta information for your entries and to apply received delta information. + +- **[Errors In Delta Propagation](errors_in_delta_propagation.html)** + + This topic lists the errors that can occur when using delta propagation. + +- **[Delta Propagation Example](delta_propagation_example.html)** + + This topic provides an example of delta propagation. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/delta_propagation/delta_propagation_example.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/delta_propagation/delta_propagation_example.html.md.erb b/geode-docs/developing/delta_propagation/delta_propagation_example.html.md.erb new file mode 100644 index 0000000..075b32e --- /dev/null +++ b/geode-docs/developing/delta_propagation/delta_propagation_example.html.md.erb @@ -0,0 +1,113 @@ +--- +title: Delta Propagation Example +--- + +This topic provides an example of delta propagation. + +<a id="delta_propagation_example__section_F6700EB7F305462D9F0DEE3080497A14"></a> +In this example, the feeder client is connected to the first server, and the receiver client is connected to the second. The servers are peers to each other. + +<img src="../../images/DeltaPropagation-3.gif" id="delta_propagation_example__image_E20DAE5925C44EBDA03D1AC8325D1122" class="image" /> + +The example demonstrates the following operations: + +1. In the Feeder client, the application updates the entry object and puts the entry. In response to the `put`, Geode calls `hasDelta`, which returns true, so Geode calls `toDelta` and forwards the extracted delta to the server. If `hasDelta` returned false, Geode would distribute the full entry value. +2. In Server1, Geode applies the delta to the cache, distributes the received delta to the server's peers, and forwards it to any other clients with interest in the entry (there are no other clients to Server1 in this example) +3. In Server2, Geode applies the delta to the cache and forwards it to its interested clients, which in this case is just the Receiver client. + +<a id="delta_propagation_example__section_185444FC51FB467587A62DFEC07C9C7D"></a> + +This example shows the basic approach to programming a `Delta` implementation. + +``` pre +package delta; + +import org.apache.geode.Delta; +import org.apache.geode.InvalidDeltaException; +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.io.Serializable; + +/** + * Sample implementation of Delta + * + * @author GemStone Systems, Inc. + * @since 6.1 + */ +public class SimpleDelta implements Delta, Serializable { + +// Original object fields + private int intVal; + private double doubleVal; + + // Added for delta - one boolean per field to track changed status + private transient boolean intFldChd = false; + private transient boolean dblFldChd = false; + + public SimpleDelta(){} + + public SimpleDelta(int i, double d){ + this.intVal = i; + this.doubleVal = d; + } + + public boolean hasDelta() { + return this.intFldChd || this.dblFldChd; + } + + public void toDelta(DataOutput out) throws IOException { + System.out.println("Extracting delta from " + this.toString()); + // Write information on what has changed to the + // data stream, so fromDelta knows what it's getting + out.writeBoolean(intFldChd); + if (intFldChd) { + // Write just the changes into the data stream + out.writeInt(this.intVal); + // Once the delta information is written, reset the delta status field + this.intFldChd = false; + System.out.println(" Extracted delta from field 'intVal' = " + + this.intVal); + } + out.writeBoolean(dblFldChd); + if (dblFldChd) { + out.writeDouble(this.doubleVal); + this.dblFldChd = false; + System.out.println(" Extracted delta from field 'doubleVal' = " + + this.doubleVal); + } + } + + public void fromDelta(DataInput in) throws IOException, InvalidDeltaException { + System.out.println("Applying delta to " + this.toString()); + // For each field, read whether there is a change + if (in.readBoolean()) { + // Read the change and apply it to the object + this.intVal = in.readInt(); + System.out.println(" Applied delta to field 'intVal' = " + + this.intVal); + } + if (in.readBoolean()) { + this.doubleVal = in.readDouble(); + System.out.println(" Applied delta to field 'doubleVal' = " + + this.doubleVal); + } + } + // In the setter methods, add setting of delta-related + // fields indicating what has changed + public void setIntVal(int anIntVal) { + this.intFldChd = true; + this.intVal = anIntVal; + } + + public void setDoubleVal(double aDoubleVal) { + this.dblFldChd = true; + this.doubleVal = aDoubleVal; + } + + public String toString() { + return "SimpleDelta [ hasDelta = " + hasDelta() + ", intVal = " + + this.intVal + ", doubleVal = {" + this.doubleVal + "} ]"; + } +} +``` http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/delta_propagation/delta_propagation_properties.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/delta_propagation/delta_propagation_properties.html.md.erb b/geode-docs/developing/delta_propagation/delta_propagation_properties.html.md.erb new file mode 100644 index 0000000..c95fab4 --- /dev/null +++ b/geode-docs/developing/delta_propagation/delta_propagation_properties.html.md.erb @@ -0,0 +1,79 @@ +--- +title: Delta Propagation Properties +--- + +This topic describes the properties that can be used to configure delta propagation. + +Delta propagation properties can be configured through the API and through the `gemfire.properties` and `cache.xml` files. + +## <a id="delta_propagation_properties__section_561D6DA876E24469B7536E98AB12F676" class="no-quick-link"></a>delta-propagation + +A `gemfire.properties` boolean that enables or disables delta propagation. When false, full entry values are sent for every update. The default setting is true, which enables delta propagation. + +Disable delta propagation as follows: + +- `gemfire.properties`: + + ``` pre + delta-propagation=false + ``` + +- API: + + ``` pre + Properties props = new Properties(); + props.setProperty("delta-propagation", false); + this.cache = new ClientCacheFactory(props).create(); + ``` + +## <a id="delta_propagation_properties__section_7D4590512D1548FD94F81C8726A2CA44" class="no-quick-link"></a>cloning-enabled + +A region attributes boolean that affects how `fromDelta` applies deltas to the local cache. When true, the updates are applied to a clone of the value and then the clone is saved to the cache. When false, the value is modified in place in the cache. The default value is false. + +Exceptions to this behavior: + +- If the `Cache` attribute `copy-on-read` is true, cloning is enabled, regardless of what this attribute is set to. +- If the `Region` attribute `off-heap` is true, cloning is enabled, regardless of what this attribute is set to. + +Cloning can be expensive, but it ensures that the new object is fully initialized with the delta before any application code sees it. + +When cloning is enabled, by default Geode does a deep copy of the object, using serialization. You may be able to improve performance by implementing `java.lang.Cloneable` and then implementing the `clone` method, making a deep copy of anything to which a delta may be applied. The goal is to reduce significantly the overhead of copying the object while still retaining the isolation needed for your deltas. + +Without cloning: + +- It is possible for application code to read the entry value as it is being modified, possibly seeing the value in an intermediate, inconsistent state, with just part of the delta applied. You may choose to resolve this issue by having your application code synchronize on reads and writes. +- Geode loses any reference to the old value because the old value is transformed in place into the new value. Because of this, your `CacheListener` sees the same new value returned for `EntryEvent.getOldValue` and `EntryEvent.getNewValue` . +- Exceptions thrown from `fromDelta` may leave your cache in an inconsistent state. Without cloning, any interruption of the delta application could leave you with some of the fields in your cached object changed and others unchanged. If you do not use cloning, keep this in mind when you program your error handling in your `fromDelta` implementation. + +With cloning: + +- The `fromDelta` method generates more garbage in memory. +- Performance is reduced. + +Enable cloning as follows: + +- `cache.xml`: + + ``` pre + <region name="region_with_cloning"> + <region-attributes refid="REPLICATE" cloning-enabled="true"> + </region-attributes> + </region> + ``` + +- API: + + ``` pre + RegionFactory rf = cache.createRegionFactory(REPLICATE); + rf.setCloningEnabled(true); + custRegion = rf.create("customer"); + ``` + +- gfsh: + + ``` pre + gfsh>create region --name="region_with_cloning" --type=REPLICATE + --enable-cloning=true + ``` + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/delta_propagation/errors_in_delta_propagation.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/delta_propagation/errors_in_delta_propagation.html.md.erb b/geode-docs/developing/delta_propagation/errors_in_delta_propagation.html.md.erb new file mode 100644 index 0000000..f6441e8 --- /dev/null +++ b/geode-docs/developing/delta_propagation/errors_in_delta_propagation.html.md.erb @@ -0,0 +1,18 @@ +--- +title: Errors In Delta Propagation +--- + +This topic lists the errors that can occur when using delta propagation. + +<a id="errors_in_delta_propagation__section_877AC61D691C44078A782683F90D169B"></a> +Errors in delta propagation fall into two categories based on how they are handled by the system: + +- Problems applying the delta that can be remedied by requesting the full value in place of the delta. Your `put` operation does not see errors or exceptions related to this type of delta propagation failure. The system automatically does a full value distribution from the sender to the receiver where the problem occurs. This type of error includes: + - Unavailable entry value in the receiving cache, either because the entry is missing or its value is null. In both cases, there is nothing to apply the delta to and the full value must be sent. This is most likely to occur if you destroy or invalidate your entries locally, either through application calls or through configured actions like eviction or entry expiration. + - `InvalidDeltaException` thrown by `fromDelta` method, programmed by you. This exception enables you to avoid applying deltas that would violate data consistency checks or other application requirements. + - Any error applying the delta in a client in server-to-client propagation. The client logs a warning in addition to retrieving the full value from the server. +- Problems creating or distributing the delta that cannot be fixed by distributing the full value. In these cases, your `put` operation fails with an exception. This type of error includes: + - Error or exception in `hasDelta` or `toDelta`. + - Error or exception in a server or peer receiver that fall outside of the situations described above in the first category. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/delta_propagation/how_delta_propagation_works.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/delta_propagation/how_delta_propagation_works.html.md.erb b/geode-docs/developing/delta_propagation/how_delta_propagation_works.html.md.erb new file mode 100644 index 0000000..a37bd8a --- /dev/null +++ b/geode-docs/developing/delta_propagation/how_delta_propagation_works.html.md.erb @@ -0,0 +1,52 @@ +--- +title: How Delta Propagation Works +--- + +Delta propagation reduces the amount of data you send over the network. You do this by only sending the change, or delta, information about an object, instead of sending the entire changed object. If you do not use cloning when applying the deltas, you can also expect to generate less garbage in your receiving JVMs. + +<a id="how_delta_propagation_works__section_78D584B3FFD04D1D9BA83203FF2B55A9"></a> +In most distributed data management systems, the data stored in the system tends to be created once and then updated frequently. These updates are sent to other members for event propagation, redundancy management, and cache consistency in general. Tracking only the changes in an updated object and sending only the deltas mean lower network transmission costs and lower object serialization/deserialization costs. Performance improvements can be significant, especially when changes to an object are small relative to its overall size. + +<a id="how_delta_propagation_works__section_ABE3589920D6477BBB2223A583AF169A"></a> + +Geode propagates object deltas using methods that you program. The methods are in the `Delta` interface, which you implement in your cached objects' classes. If any of your classes are plain old Java objects, you need to wrap them for this implementation. + +This figure shows delta propagation for a change to an entry with key, k, and value object, v. + +<img src="../../images/DeltaPropagation-1.gif" id="how_delta_propagation_works__image_06A25185C31548669423DDA3136B1851" class="image" /> + +1. **`get` operation**. The `get` works as usual: the cache returns the full entry object from the local cache or, if it isn't available there, from a remote cache or from a loader. +2. **update methods**. You need to add code to the object's update methods so that they save delta information for object updates, in addition to the work they were already doing. +3. **`put` operation**. The `put` works as usual in the local cache, using the full value, then calls `hasDelta` to see if there are deltas and `toDelta` to serialize the information. Distribution is the same as for full values, according to member and region configuration. +4. **receipt of delta at remote member**. `fromDelta` extracts the delta information that was serialized by `toDelta` and applies it to the object in the local cache. The delta is applied directly to the existing value or to a clone, depending on how you configure it for the region. +5. **additional distributions**. As with full distributions, receiving members forward the delta according to their configurations and connections to other members. For example, if VM1 is a client and VM2 is a server, VM2 forwards the delta to its peers and its other clients as needed. Receiving members do not recreate the delta; `toDelta` is only called in the originating member. + +## <a id="how_delta_propagation_works__section_25EC5BE960F0402DAEDBE0A5A6589ACA" class="no-quick-link"></a>General Characteristics of Delta Propagation + +To use the delta propagation feature, all updates on a key in a region must have value types that implement the `Delta` interface. You cannot mix object types for an entry key where some of the types implement delta and some do not. This is because, when a type implementing the delta interface is received for an update, the existing value for the key is cast to a `Delta` type to apply the received delta. If the existing type does not also implement the `Delta` interface, the operation throws a `ClassCastException`. + +**Note:** Only the object itself being placed in the cache can implement the `Delta` interface and propagate changes. Any sub-objects of the cache object do not propagate their changes. + +Sometimes `fromDelta` cannot be invoked because there is no object to apply the delta to in the receiving cache. When this happens, the system automatically does a full value distribution to the receiver. These are the possible scenarios: +1. If the system can determine beforehand that the receiver does not have a local copy, it sends the initial message with the full value. This is possible when regions are configured with no local data storage, such as with the region shortcut settings `PARTITION_PROXY` and `REPLICATE_PROXY`. These configurations are used to accomplish things like provide data update information to listeners and to pass updates forward to clients. +2. In less obvious cases, such as when an entry has been locally deleted, first the delta is sent, then the receiver requests a full value and that is sent. Whenever the full value is received, any further distributions to the receiver's peers or clients uses the full value. + +Geode also does not propagate deltas for: + +- Transactional commit +- The `putAll` operation +- JVMs running Geode versions that do not support delta propagation (6.0 and earlier) + +## <a id="how_delta_propagation_works__section_F4A102A74530429F87BEA53C90D5CCFB" class="no-quick-link"></a>Supported Topologies and Limitations + +The following topologies support delta propagation (with some limitations): + +- **Peer-to-peer**. Geode system members distribute and receive entry changes using delta propagation, with these requirements and caveats: + - Regions must be partitioned or have their scope set to `distributed-ack` or `global`. The region shortcut settings for distributed regions use `distributed-ack` `scope`. Delta propagation does not work for regions with `distributed-no-ack` `scope` because the receiver could not recover if an exception occurred while applying the delta. + - For partitioned regions, if a receiving peer does not hold the primary or a secondary copy of the entry, but still requires a value, the system automatically sends the full value. + - To receive deltas, a region must be non-empty. The system automatically sends the full value to empty regions. Empty regions can send deltas. +- **Client/server**. Geode clients can always send deltas to the servers, and servers can usually sent deltas to clients. These configurations require the servers to send full values to the clients, instead of deltas: + - When the client's `gemfire.properties` setting `conflate-events` is set to true, the servers send full values for all regions. + - When the server region attribute `enable-subscription-conflation` is set to true and the client `gemfire.properties` setting `conflate-events` is set to `server`, the servers send full values for the region. + - When the client region is configured with the `PROXY` client region shortcut setting (empty client region), servers send full values. + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/delta_propagation/implementing_delta_propagation.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/delta_propagation/implementing_delta_propagation.html.md.erb b/geode-docs/developing/delta_propagation/implementing_delta_propagation.html.md.erb new file mode 100644 index 0000000..7727532 --- /dev/null +++ b/geode-docs/developing/delta_propagation/implementing_delta_propagation.html.md.erb @@ -0,0 +1,24 @@ +--- +title: Implementing Delta Propagation +--- + +By default, delta propagation is enabled in your distributed system. When enabled, delta propagation is used for objects that implement `org.apache.geode.Delta`. You program the methods to store and extract delta information for your entries and to apply received delta information. + +<a id="implementing_delta_propagation__section_877AC61D691C44078A782683F90D169B"></a> +Use the following procedure to implement delta propagation in your distributed system. + +1. Study your object types and expected application behavior to determine which regions can benefit from using delta propagation. Delta propagation does not improve performance for all data and data modification scenarios. See [When to Avoid Delta Propagation](when_to_use_delta_prop.html#when_to_use_delta_prop). +2. For each region where you are using delta propagation, choose whether to enable cloning using the delta propagation property `cloning-enabled`. Cloning is disabled by default. See [Delta Propagation Properties](delta_propagation_properties.html#delta_propagation_properties). +3. If you do not enable cloning, review all associated listener code for dependencies on `EntryEvent.getOldValue`. Without cloning, Geode modifies the entry in place and so loses its reference to the old value. For delta events, the `EntryEvent` methods `getOldValue` and `getNewValue` both return the new value. +4. For every class where you want delta propagation, implement `org.apache.geode.Delta` and update your methods to support delta propagation. Exactly how you do this depends on your application and object needs, but these steps describe the basic approach: + 1. If the class is a plain old Java object (POJO), wrap it for this implementation and update your code to work with the wrapper class. + 2. Define as transient any extra object fields that you use to manage delta state. This can help performance when the full object is distributed. Whenever standard Java serialization is used, the transient keyword indicates to Java to not serialize the field. + 3. Study the object contents to decide how to handle delta changes. Delta propagation has the same issues of distributed concurrency control as the distribution of full objects, but on a more detailed level. Some parts of your objects may be able to change independent of one another while others may always need to change together. Send deltas large enough to keep your data logically consistent. If, for example, field A and field B depend on each other, then your delta distributions should either update both fields or neither. As with regular updates, the fewer producers you have on a data region, the lower your likelihood of concurrency issues. + 4. In the application code that puts entries, put the fully populated object into the local cache. Even though you are planning to send only deltas, errors on the receiving end could cause Geode to request the full object, so you must provide it to the originating put method. Do this even in empty producers, with regions configured for no local data storage. This usually means doing a get on the entry unless you are sure it does not already exist anywhere in the distributed region. + 5. Change each field's update method to record information about the update. The information must be sufficient for `toDelta` to encode the delta and any additional required delta information when it is invoked. + 6. Write `hasDelta` to report on whether a delta is available. + 7. Write `toDelta` to create a byte stream with the changes to the object and any other information `fromDelta` will need to apply the changes. Before returning from `toDelta`, reset your delta state to indicate that there are no delta changes waiting to be sent. + 8. Write `fromDelta` to decode the byte stream that `toDelta` creates and update the object. + 9. Make sure you provide adequate synchronization to your object to maintain a consistent object state. If you do not use cloning, you will probably need to synchronize on reads and writes to avoid reading partially written updates from the cache.This synchronization might involve `toDelta`, `fromDelta`, `toData`, `fromData`, and other methods that access or update the object. Additionally, your implementation should take into account the possibility of concurrent invocations of `fromDelta` and one or more of the object's update methods. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/delta_propagation/when_to_use_delta_prop.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/delta_propagation/when_to_use_delta_prop.html.md.erb b/geode-docs/developing/delta_propagation/when_to_use_delta_prop.html.md.erb new file mode 100644 index 0000000..47de0ba --- /dev/null +++ b/geode-docs/developing/delta_propagation/when_to_use_delta_prop.html.md.erb @@ -0,0 +1,17 @@ +--- +title: When to Avoid Delta Propagation +--- + +Generally, the larger your objects and the smaller the deltas, the greater the benefits of using delta propagation. Partitioned regions with higher redundancy levels generally benefit more from delta propagation. However, in some application scenarios, delta propagation does not show any significant benefits. On occasion it results in performance degradation. + +<a id="when_to_use_delta_prop__section_83BA84BB08194FC58F2BCE149AA0F0EC"></a> +By default, delta propagation is enabled in your distributed system. + +These are the main factors that can reduce the performance benefits of using delta propagation: + +- The added costs of deserializing your objects to apply deltas. Applying a delta requires the entry value to be deserialized. Once this is done, the object is stored back in the cache in deserialized form. This aspect of delta propagation only negatively impacts your system if your objects are not already being deserialized for other reasons, such as for indexing and querying or for listener operations. Once stored in deserialized form, there are reserialization costs for operations that send the object outside of the member, like distribution from a gateway sender, values sent in response to `netSearch` or client requests, and storage to disk. The more operations that require reserialization, the higher the overhead of deserializing the object. As with all serialization efforts, you can improve performance in serialization and deserialization by providing custom implementations of `DataSerializable` for your objects. +- Cloning when applying the delta. Using cloning can affect performance and generates extra garbage. Not using cloning is risky however, as you are modifying cached values in place. Without cloning, make sure you synchronize your entry access to keep your cache from becoming inconsistent. +- Problems applying the delta that cause the system to go back to the originator for the full entry value. When this happens, the overall operation costs more than sending the full entry value in the first place. This can be additionally aggravated if your delta is sent to a number of recipients, all or most of them request a full value, and the full value send requires the object to be serialized. +- Disk I/O costs associated with overflow regions. If you use eviction with overflow to disk, on-disk values must be brought into memory in order to apply the delta. This is much more costly than just removing the reference to the disk copy, as you would do with a full value distribution into the cache. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/distributed_regions/chapter_overview.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/distributed_regions/chapter_overview.html.md.erb b/geode-docs/developing/distributed_regions/chapter_overview.html.md.erb new file mode 100644 index 0000000..ce33ee2 --- /dev/null +++ b/geode-docs/developing/distributed_regions/chapter_overview.html.md.erb @@ -0,0 +1,27 @@ +--- +title: Distributed and Replicated Regions +--- + +In addition to basic region management, distributed and replicated regions include options for things like push and pull distribution models, global locking, and region entry versions to ensure consistency across Geode members. + +- **[How Distribution Works](../../developing/distributed_regions/how_distribution_works.html)** + + To use distributed and replicated regions, you should understand how they work and your options for managing them. + +- **[Options for Region Distribution](../../developing/distributed_regions/choosing_level_of_dist.html)** + + You can use distribution with and without acknowledgment, or global locking for your region distribution. Regions that are configured for distribution with acknowledgment can also be configured to resolve concurrent updates consistently across all Geode members that host the region. + +- **[How Replication and Preloading Work](../../developing/distributed_regions/how_replication_works.html)** + + To work with replicated and preloaded regions, you should understand how their data is initialized and maintained in the cache. + +- **[Configure Distributed, Replicated, and Preloaded Regions](../../developing/distributed_regions/managing_distributed_regions.html)** + + Plan the configuration and ongoing management of your distributed, replicated, and preloaded regions, and configure the regions. + +- **[Locking in Global Regions](../../developing/distributed_regions/locking_in_global_regions.html)** + + In global regions, the system locks entries and the region during updates. You can also explicitly lock the region and its entries as needed by your application. Locking includes system settings that help you optimize performance and locking behavior between your members. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/distributed_regions/choosing_level_of_dist.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/distributed_regions/choosing_level_of_dist.html.md.erb b/geode-docs/developing/distributed_regions/choosing_level_of_dist.html.md.erb new file mode 100644 index 0000000..f48aaeb --- /dev/null +++ b/geode-docs/developing/distributed_regions/choosing_level_of_dist.html.md.erb @@ -0,0 +1,19 @@ +--- +title: Options for Region Distribution +--- + +You can use distribution with and without acknowledgment, or global locking for your region distribution. Regions that are configured for distribution with acknowledgment can also be configured to resolve concurrent updates consistently across all Geode members that host the region. + +<a id="choosing_level_of_dist__section_F2528B151DD54CEFA05C4BA655BCF016"></a> +Each distributed region must have the same scope and concurrency checking setting throughout the distributed system. + +Distributed scope is provided at three levels: + +- **distributed-no-ack**. Distribution operations return without waiting for a response from other caches. This scope provides the best performance and uses the least amount of overhead, but it is also most prone to having inconsistencies caused by network problems. For example, a temporary disruption of the network transport layer could cause a failure in distributing updates to a cache on a remote machine, while the local cache continues being updated. +- **distributed-ack**. Distribution waits for acknowledgment from other caches before continuing. This is slower than `distributed-no-ack`, but covers simple communication problems such as temporary network disruptions. + + In systems where there are many `distributed-no-ack` operations, it is possible for `distributed-ack` operations to take a long time to complete. The distributed system has a configurable time to wait for acknowledgment to any `distributed-ack` message before sending alerts to the logs about a possible problem with the unresponsive member. No matter how long the wait, the sender keeps waiting in order to honor the distributed-ack region setting. The `gemfire.properties` attribute governing this is `ack-wait-threshold`. + +- **global**. Entries and regions are automatically locked across the distributed system during distribution operations. All load, create, put, invalidate, and destroy operations on the region and its entries are performed with a distributed lock. The global scope enforces strict consistency across the distributed system, but it is the slowest mechanism for achieving consistency. In addition to the implicit locking performed by distribution operations, regions with global scope and their contents can be explicitly locked through the application APIs. This allows applications to perform atomic, multi-step operations on regions and region entries. + + http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/ccc2fbda/geode-docs/developing/distributed_regions/how_distribution_works.html.md.erb ---------------------------------------------------------------------- diff --git a/geode-docs/developing/distributed_regions/how_distribution_works.html.md.erb b/geode-docs/developing/distributed_regions/how_distribution_works.html.md.erb new file mode 100644 index 0000000..bbc7522 --- /dev/null +++ b/geode-docs/developing/distributed_regions/how_distribution_works.html.md.erb @@ -0,0 +1,31 @@ +--- +title: How Distribution Works +--- + +To use distributed and replicated regions, you should understand how they work and your options for managing them. + +<a id="how_distribution_works__section_2F892A4987C547E68CA78067133C2C2C"></a> +**Note:** +The management of replicated and distributed regions supplements the general information for managing data regions provided in [Basic Configuration and Programming](../../basic_config/book_intro.html). See also `org.apache.geode.cache.PartitionAttributes`. + +A distributed region automatically sends entry value updates to remote caches and receives updates from them. + +- Distributed entry updates come from the `Region` `put` and `create` operations (the creation of an entry with a non-null value is seen as an update by remote caches that already have the entry key). Entry updates are distributed selectively - only to caches where the entry key is already defined. This provides a pull model of distribution, compared to the push model that you get with replication. +- Distribution alone does not cause new entries to be copied from remote caches. +- A distributed region shares cache loader and cache writer application event handler plug-ins across the distributed system. + +In a distributed region, new and updated entry values are automatically distributed to remote caches that already have the entries defined. + +**Step 1:** The application updates or creates the entry. At this point, the entry in the M1 cache may not yet exist. + +<img src="../../images_svg/distributed_how_1.svg" id="how_distribution_works__image_40EFE6E95E6945A1B08A68508ECBCC60" class="image" /> + +**Step 2:** The new value is automatically distributed to caches holding the entry. + +<img src="../../images_svg/distributed_how_2.svg" id="how_distribution_works__image_AF8A3ADEB5D94E20B101FDA92BF6D002" class="image" /> + +**Step 3:** The entry's value is the same throughout the distributed system. + +<img src="../../images_svg/distributed_how_3.svg" id="how_distribution_works__image_5B1F06B54C9047E28A8C8673D1D5BD27" class="image" /> + +
