RE: Best practice for class versioning: marshaller error
Got it, thanks for the info. Thanks, Calvin -Original Message- From: dkarachentsev [mailto:dkarachent...@gridgain.com] Sent: Monday, July 16, 2018 9:57 PM To: user@ignite.apache.org Subject: RE: Best practice for class versioning: marshaller error Hi Calvin, > Can I assume that BinaryMarshaller won't be used for any object embedded > inside GridCacheQueryResponse? Yes, because Binary can fallback to Optimized, but not vice versa. > If I am correct, do you have any suggestion on how I can avoid this type > of issue? Probably you need to avoid using incorrectly serialized objects, or implement your own Externalizable, when you will do manual serializing/deserializing of such fields. Thanks! -Dmitry -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices. These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request. Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.
RE: Best practice for class versioning: marshaller error
Hi Calvin, > Can I assume that BinaryMarshaller won't be used for any object embedded > inside GridCacheQueryResponse? Yes, because Binary can fallback to Optimized, but not vice versa. > If I am correct, do you have any suggestion on how I can avoid this type > of issue? Probably you need to avoid using incorrectly serialized objects, or implement your own Externalizable, when you will do manual serializing/deserializing of such fields. Thanks! -Dmitry -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Best practice for class versioning: marshaller error
601) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3.1.1.1 at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) ~[?:1.8.0_74] at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3.1.1 at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:545) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365) ~[?:1.8.0_74] at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:227) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1783) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3.1 at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1962) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 3 at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.unmarshalCollection0(GridCacheQueryResponse.java:189) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 2 at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.finishUnmarshal(GridCacheQueryResponse.java:162) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1526) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] <<<<<<<<<<<<<<<<< Step 1 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:574) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] -Original Message- From: slava.koptilin [mailto:slava.kopti...@gmail.com] Sent: Friday, June 29, 2018 1:04 AM To: user@ignite.apache.org Subject: RE: Best practice for class versioning: marshaller error Hello, In that case, only 'createdTime' fall back to OptimizedMarshaller. Thanks, Slava. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices. These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request. Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.
RE: Best practice for class versioning: marshaller error
Hello, In that case, only 'createdTime' fall back to OptimizedMarshaller. Thanks, Slava. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Best practice for class versioning: marshaller error
Thanks, Dmitry.
>>2-3. In your case, you have and java.time.Ser in one of the fields of your
>>POJO (or maybe inside of depended object), and it is Externalizable. In such
>>case BinaryMarshalelr falls back to OptimizedMarshaller with all the issues.
>>Try to remove it from your POJOs or make transient.
Given that I have this class:
class NewOrderRequest {
int securityId;
Instant createdTime;
}
Instant has a writeReplace method that returns java.time.Ser.
Is the whole 'NewOrderRequest' fall back to OptimizedMarshaller? Or only
'createdTime' fall back to OptimizedMarshaller.
-Calvin
-Original Message-
From: dkarachentsev [mailto:dkarachent...@gridgain.com]
Sent: Thursday, June 28, 2018 2:59 PM
To: user@ignite.apache.org
Subject: RE: Best practice for class versioning: marshaller error
Hi Calvin,
1. Enlist I mean that if you want, for example, to get to see what fields
present in BinaryObject. In other words, if you want to work with
BinaryObject directly. For POJO serialization/deserialization this should
not be and issue at all.
2-3. In your case, you have and java.time.Ser in one of the fields of your
POJO (or maybe inside of depended object), and it is Externalizable. In such
case BinaryMarshalelr falls back to OptimizedMarshaller with all the issues.
Try to remove it from your POJOs or make transient.
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
The content of this communication is intended for the recipient and is subject
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon
request.
Please consider before printing. CLSA is ISO14001 certified and committed to
reducing its impact on the environment.
RE: Best practice for class versioning: marshaller error
Hi Calvin, 1. Enlist I mean that if you want, for example, to get to see what fields present in BinaryObject. In other words, if you want to work with BinaryObject directly. For POJO serialization/deserialization this should not be and issue at all. 2-3. In your case, you have and java.time.Ser in one of the fields of your POJO (or maybe inside of depended object), and it is Externalizable. In such case BinaryMarshalelr falls back to OptimizedMarshaller with all the issues. Try to remove it from your POJOs or make transient. Thanks! -Dmitry -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Best practice for class versioning: marshaller error
Hi Dmitry, Thanks for your response! I want to confirm my understanding - If I disable compact footer and add your POJOs in BinaryConfiguration, I will be deserialize back the serialized object after I stop grid and removed the local files that hold the marshaller cache (I believe they are located in the tmp folder). Questions: 1. >> Second case, if you by some reason want to use BinaryObject directly. You should be aware that fact you can read and add fields, but you will not be able to enlist them. Again, hopefully, you can solve this with BinaryConfiguration I don't get the 'enlist' part in this sentence. Does that mean I cannot turn them back into POJO even if I list my POJO in BinaryConfiguration? Or this has something to do with BinaryObject. 2. Regarding BinaryObject, I have a regular POJO user class that does not implement Binarylizable and Externalizable, and has no 'writeObject' and 'readObject' method (BinaryContext.canUseReflectiveSerializer). I see that its BinaryClassDescriptor has a mode of BinaryWriteMode.OBJECT, not BinaryWriteMode.BINARY_OBJ. Does not mean I cannot use it as BinaryObject? From reading the doc, looks like all sample objects will be serialized as BinaryObject automatically (https://apacheignite.readme.io/docs/binary-marshaller). However, that is not what I am seeing in the code. 3. The error log shows that OptimizedMarshaller is being called. Since all my classes are neither Binarylizable nor Externalizable, does that mean the marshalling error is related to versioning of an Ignite class? Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$CacheClassLoader@3bc180c9, err=Unexpected error occurred during unmarshalling of an instance of the class: java.time.Ser. Check that all nodes are running the same version of Ignite and that all nodes have GridOptimizedMarshaller configured with identical optimized classes lists, if any (see setClassNames and setClassNamesPath methods). If your serialized classes implement java.io.Externalizable interface, verify that serialization logic is correct.] at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:236) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] at Please note that I ask these questions because I need to look into an issue that happened some time ago. I have the log with me, but I am not able to reproduce the issue. That's why I try to understand more. Thanks, Calvin -Original Message- From: dkarachentsev [mailto:dkarachent...@gridgain.com] Sent: Wednesday, June 27, 2018 8:52 PM To: user@ignite.apache.org Subject: Re: Best practice for class versioning: marshaller error Hi Calvin, BinaryMarshaller can solve that issue with involving a few more. First of all, you will need to disable compact footer to let each BinaryObject has it's schema in footer. If you need just put/get POJOs everything will be fine. But you need to enlist your POJO in BinaryConfiguration [1], because Ignite identifies type with typeId, that is by default hash from full class name. So each BinaryObject keeps only typeId. To find proper class to which it should be deserialized, Ignite needs mapping class name -> typeId. This mapping keeps in marshaller cache that holds in memory and in local files. Anyway you can stop grid, remove Ignite files and ran into issue that you cannot deserialize object, because Ignite cannot find class name for some typeId. And as I said, you need to enlist this class in BinaryConfiguration, that Ignite will read on start and fill marshaller cache. Second case, if you by some reason want to use BinaryObject directly. You should be aware that fact you can read and add fields, but you will not be able to enlist them. Because here used the same approach: field name converts to fieldId. Mapping fieldName -> fieldId is called meta info and stored in local meta cache. Again, hopefully, you can solve this with BinaryConfiguration. In summary, BinaryMarshaller is OK with different schema, but you need disable compact footer and add your POJOs in BinaryConfiguration. [1] https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/BinaryConfiguration.html Thanks! -Dmitry -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices. These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request. Please consider before printing. CLSA is ISO14001 certified and committed to reducing its impact on the environment.
Re: Best practice for class versioning: marshaller error
Hi Calvin, BinaryMarshaller can solve that issue with involving a few more. First of all, you will need to disable compact footer to let each BinaryObject has it's schema in footer. If you need just put/get POJOs everything will be fine. But you need to enlist your POJO in BinaryConfiguration [1], because Ignite identifies type with typeId, that is by default hash from full class name. So each BinaryObject keeps only typeId. To find proper class to which it should be deserialized, Ignite needs mapping class name -> typeId. This mapping keeps in marshaller cache that holds in memory and in local files. Anyway you can stop grid, remove Ignite files and ran into issue that you cannot deserialize object, because Ignite cannot find class name for some typeId. And as I said, you need to enlist this class in BinaryConfiguration, that Ignite will read on start and fill marshaller cache. Second case, if you by some reason want to use BinaryObject directly. You should be aware that fact you can read and add fields, but you will not be able to enlist them. Because here used the same approach: field name converts to fieldId. Mapping fieldName -> fieldId is called meta info and stored in local meta cache. Again, hopefully, you can solve this with BinaryConfiguration. In summary, BinaryMarshaller is OK with different schema, but you need disable compact footer and add your POJOs in BinaryConfiguration. [1] https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/BinaryConfiguration.html Thanks! -Dmitry -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
