Re: Cassandra cache store [IGNITE-1371]
Hi guys, Just a short update for Cassandra persistent store implementation: 1) We switched to the newest version of Cassandra 3.3 2) Unit tests now using embedded Cassandra starting in the same process with Ignite cluster. 3) All the latest changes from Ignite 1.6 upstream were merged into repository: https://github.com/irudyak/ignite 4) Current implementation is available in branch "ignite-1371" -- View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p8313.html Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: Cassandra cache store [IGNITE-1371]
Hi guys, I am quite busy now. It looks like I'll just start implementing PersistenceCallback in a month. -- View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7827.html Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: Cassandra cache store [IGNITE-1371]
I'm quite sure that 1.6 release won't be ready in a month. So you have a plenty of time to finish your ticket ;) On 2/19/2016 10:54 AM, irudyak wrote: Well, it just depends on how soon you are going to publish release 1.6. It will take about 1 month for me to implement PersistenceCallback. -- View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7466.html Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: Cassandra cache store [IGNITE-1371]
Well, it just depends on how soon you are going to publish release 1.6. It will take about 1 month for me to implement PersistenceCallback. -- View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7466.html Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: Cassandra cache store [IGNITE-1371]
I think it makes sense to make this feature the one that drives the next release. There is a discussion started by Yakov on this list named "Ignite 1.6 release timelines". https://mail-archives.apache.org/mod_mbox/ignite-dev/201602.mbox/%3CCAGcMBHiS-9SaqURK8pKQ98VKh-Ynoq_y723%2BotLDuNJJAjKgow%40mail.gmail.com%3E Igor, could you ask to consider this ticket for the next release as well answering in that thread? -- Denis On 2/18/2016 10:37 AM, Dmitriy Setrakyan wrote: Would be nice to get it into the next release. On Wed, Feb 17, 2016 at 10:40 PM, irudyakwrote: Actually the idea of PersistenceCallback interface looks good. I'll try to think how it's better to implement CassandraData class. -- View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7436.html Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: Cassandra cache store [IGNITE-1371]
Would be nice to get it into the next release. On Wed, Feb 17, 2016 at 10:40 PM, irudyakwrote: > Actually the idea of PersistenceCallback interface looks good. I'll try to > think how it's better to implement CassandraData class. > > > > -- > View this message in context: > http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7436.html > Sent from the Apache Ignite Developers mailing list archive at Nabble.com. >
Re: Cassandra cache store [IGNITE-1371]
Actually the idea of PersistenceCallback interface looks good. I'll try to think how it's better to implement CassandraData class. -- View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Cassandra-cache-store-IGNITE-1371-tp6880p7436.html Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: Cassandra cache store [IGNITE-1371]
I like Val suggestion. Also in this case we could move external things like Kryo into separate module/dependency (for those who do not want to depend on Kryo). This could be "ignite-cassandra-kryo" module that will contain smth. like "KryoPersistenceCallback". Thoughts? On Fri, Jan 29, 2016 at 10:27 AM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Alexey, Igor, > > Binary format is internal format used by Ignite, it can't be used directly > or outside of Ignite. So there is no way and no reason to have special > binary serializer in Cassandra store implementation. If user wants to save > BinaryObject as a BLOB to the store (to later load it back to Ignite), he > can set CacheConfiguration.setStoreKeepBinary(true) flag and cache will > give BinaryObject to the store without deserializing it. > > Actually, I'm looking at the current API and I don't like > PersistenceStrategy enum and Serializer interface. I understand what you > try to achieve here and it makes sense to me, but the abstraction is not > generic enough in my opinion. We give user only two options, but what if he > needs something else? For example, he may need to add special processing > for some specific fields (make some conversions, etc.). > > I propose to change it a bit and do something like this: > > interface PersistenceCallback { > CassandraData onWrite(K key, V value); > IgniteBiTupleonRead(CassandraData); > } > > CassandraData is an entity that Cassandra should be able to write and load > (essentially, set of column values). We can reuse something from Cassandra > API here if that's possible (I'm not familiar enough with Cassandra to make > specific suggestions right away) or create our own. > > We will have a default implementation that will introspect POJOs. The one > that converts value to BLOB can be also provided out of the box. And user > is free to implement his own logic. > > All namings and API itself are arguable, of course, but I hope that idea is > clear. > > Thoughts? > > On Thu, Jan 28, 2016 at 12:41 AM, Alexey Kuznetsov < > akuznet...@gridgain.com> > wrote: > > > During review I faced the problem, that I have not enough experience to > > answer. > > > > So, it will be great, if someone who has deep experience with Binary > > marshaller could help us. > > > > --- > > > > > How about such limitations of Binary Marshaller: > > > > > 1) Fields or types with the same name hash are not allowed. > > > 2) BinaryObject format does not allow same field names on different > > levels of a class hierarchy. > > > 3) Externalizable interface is ignored by default. If BinaryObject > format > > is used, Externalizable > > > classes will be written the same way as if they were Serializable, > > without writeExternal() and > > > readExternal() methods. If for some reason this does not work for you, > > you should implement > > > Binarylizable interface for your classes, plug in a custom > > BinarySerializer or switch to the > > > OptimizedMarshaller. > > > > > There are no such limitations in Kryo and Java serialization. > > > > > The next concern is that you need *Ignite Core* module, which is rather > > huge (about 7.3MB) if > > > you want to build ETL script which will consume data persisted into > > Cassandra by Ignite Binary > > > Marshaller. > > > > > By the way, does Ignite plan to support backward compatibility for data > > persisted using Binary > > > Marshaller? I mean the situation when some objects were persisted into > > Cassandra using old > > > version of Binary Marshaller and then Ignite cluster was upgraded to > new > > version. Kryo and > > > Java serialization handles this situation and provides backward > > compatibility. > > > > > May be it's better just to add one more serializer implementation which > > will use Binary Marshaller? > > > > > By the way are there any samples in the code how to use Binary > Marshaller > > just to > > > serialize/deserialize arbitrary object? Binary Marshaller documentation > > says that all such > > > operation performed internally inside Ignite when we using > BinaryObject, > > but in my case I am > > > interested in rather low-level serialization/deserialization API/ > > > > Regards, > > Igor Rudyak > > > > > > Hi Igor! > > > > I moved discussion to dev list. > > > > >>> The next concern is that you need *Ignite Core* module > > We could not avoid adding this module because All base cache store > classes > > declared in that module, so I think this module will *be always > imported*. > > > > As for other points, I hope community will help us. > > > > -- > > Alexey Kuznetsov > > GridGain Systems > > www.gridgain.com > > > -- Alexey Kuznetsov GridGain Systems www.gridgain.com
Re: Cassandra cache store [IGNITE-1371]
Alexey, Igor, Binary format is internal format used by Ignite, it can't be used directly or outside of Ignite. So there is no way and no reason to have special binary serializer in Cassandra store implementation. If user wants to save BinaryObject as a BLOB to the store (to later load it back to Ignite), he can set CacheConfiguration.setStoreKeepBinary(true) flag and cache will give BinaryObject to the store without deserializing it. Actually, I'm looking at the current API and I don't like PersistenceStrategy enum and Serializer interface. I understand what you try to achieve here and it makes sense to me, but the abstraction is not generic enough in my opinion. We give user only two options, but what if he needs something else? For example, he may need to add special processing for some specific fields (make some conversions, etc.). I propose to change it a bit and do something like this: interface PersistenceCallback { CassandraData onWrite(K key, V value); IgniteBiTupleonRead(CassandraData); } CassandraData is an entity that Cassandra should be able to write and load (essentially, set of column values). We can reuse something from Cassandra API here if that's possible (I'm not familiar enough with Cassandra to make specific suggestions right away) or create our own. We will have a default implementation that will introspect POJOs. The one that converts value to BLOB can be also provided out of the box. And user is free to implement his own logic. All namings and API itself are arguable, of course, but I hope that idea is clear. Thoughts? On Thu, Jan 28, 2016 at 12:41 AM, Alexey Kuznetsov wrote: > During review I faced the problem, that I have not enough experience to > answer. > > So, it will be great, if someone who has deep experience with Binary > marshaller could help us. > > --- > > > How about such limitations of Binary Marshaller: > > > 1) Fields or types with the same name hash are not allowed. > > 2) BinaryObject format does not allow same field names on different > levels of a class hierarchy. > > 3) Externalizable interface is ignored by default. If BinaryObject format > is used, Externalizable > > classes will be written the same way as if they were Serializable, > without writeExternal() and > > readExternal() methods. If for some reason this does not work for you, > you should implement > > Binarylizable interface for your classes, plug in a custom > BinarySerializer or switch to the > > OptimizedMarshaller. > > > There are no such limitations in Kryo and Java serialization. > > > The next concern is that you need *Ignite Core* module, which is rather > huge (about 7.3MB) if > > you want to build ETL script which will consume data persisted into > Cassandra by Ignite Binary > > Marshaller. > > > By the way, does Ignite plan to support backward compatibility for data > persisted using Binary > > Marshaller? I mean the situation when some objects were persisted into > Cassandra using old > > version of Binary Marshaller and then Ignite cluster was upgraded to new > version. Kryo and > > Java serialization handles this situation and provides backward > compatibility. > > > May be it's better just to add one more serializer implementation which > will use Binary Marshaller? > > > By the way are there any samples in the code how to use Binary Marshaller > just to > > serialize/deserialize arbitrary object? Binary Marshaller documentation > says that all such > > operation performed internally inside Ignite when we using BinaryObject, > but in my case I am > > interested in rather low-level serialization/deserialization API/ > > Regards, > Igor Rudyak > > > Hi Igor! > > I moved discussion to dev list. > > >>> The next concern is that you need *Ignite Core* module > We could not avoid adding this module because All base cache store classes > declared in that module, so I think this module will *be always imported*. > > As for other points, I hope community will help us. > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com >