Hey guys, About the TCCL, I need to dig deeper into how serialisation is being done now. If, at any point, we are resolving a class through the classloader of a particular class, e.g. Ignition.getClass().getClassLoader(), etc. we are using the class space of the bundle containing that class. If we are using a class from Ignite, we obviously wouldn't be able to find a custom DTO provided by a user.
In Camel we have found this problem several times and we've solved it by changing the TCCL, or using the TCCL [1] [2] to resolve classes. As I said, I need to look into this deeper and I'll have some time on Thursday, so I hope you don't mind waiting a little bit. I have already "bundle-fied" most of Ignite modules in the corresponding branch [3], so my next step is to test out the Ignite examples inside an OSGi environment (Karaf 4). [1] https://issues.apache.org/jira/browse/CAMEL-5748 [2] https://issues.apache.org/jira/browse/CAMEL-5722 [3] https://github.com/apache/ignite/tree/ignite-1527 Regards, *Raúl Kripalani* PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and Messaging Engineer http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani http://blog.raulkr.net | twitter: @raulvk On Tue, Nov 3, 2015 at 10:38 AM, Romain Gilles <romain.gil...@gmail.com> wrote: > Hi Raul, > - Do you plan to use the TCCL when marshalling in OSGi env? If yes how? > - I like the point 2. Maybe for a graph of object that come from different > packages / bundles you may have to recursively capture the package version > of the individual element of your graph. > - For the point 3 I wonder if it will not over-complexify the solution. As > a developer you will have to think about it. And it is not obvious in the > code where things are serialized or not. You may use lambda in your > application code therefore the current package become what you call a DTO > package. The current state of ignite does not enforce you to specify it for > "classical" classloading application. If you introduce this extra step for > OSGi ready application you will maybe discourage people to use OSGi. > > To comeback to our solution. We start we a strong assumption: our cluster > is homogeneous in term of code so, of course, it simplify our life :). > > BTW if you can introduce an extension point in the class resolution > mechanism it can be interesting for end user in order to allow them to > optimize it based on there specific use cases. > > Romain. > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <r...@evosent.com> a écrit : > > > Hi Andrey, > > > > Thanks for the participation in this topic. > > > > I don't like the solution to incorporate the bundle symbolic name in the > > serialised form. Nothing guarantees that the classdef will be located > under > > the same bundle in both source and target machines. We also have to take > > into account that OSGi allows for multiple versions of the same > > bundle/packages to co-exist in the same container. So it becomes more > > complex. > > > > Using the TCCL should work when serialising, but it'll probably be of no > > use when deserialising on the other end. > > > > I need to enhance Ignite to: > > > > 1. Use the TCCL when marshalling on one end. > > 2. Incorporate the package version of the class in the serialised form if > > Ignite is running in an OSGi environment. > > 3. On the receiver end, discover cache entities / DTOs in all bundles > > through a custom OSGi manifest header or the like, as I explained before. > > Package version must be taken into account. > > > > What do you think? > > > > Raúl. > > On 2 Nov 2015 17:25, "Andrey Kornev" <andrewkor...@hotmail.com> wrote: > > > > > Raul, > > > > > > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled > > is > > > the marshalling. More specifically the issue is with deserialization of > > the > > > classes that are provided by the bundles other than the Ignite bundle > > > itself. > > > > > > When the Ignite transport layer receives a message it needs to figure > out > > > how to deserialize the bytes and for that it needs to know the bundle > > that > > > provides the class to be deserailized. At this point TCCL is of no use. > > To > > > make things more complex, the class may contain other classes that come > > > from other bundles, and so on recursively. This means that each object > in > > > the hierarchy must be serialized with its bundle name (or bundle id), > so > > > that the deserializer will then be able to correctly resolve the class > > > while traversing the object hierarchy during deserialization. > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to > > plug > > > in a custom class resolver. Romain's solution was to use Kryo that does > > > provide a way to customize class resolution. It has solved the problem > of > > > capturing the bundle info and he was able to successfully run Ignite > as a > > > bundle in an OSGi container (after some repackaging and inclusion of > the > > > manifest). But Kryo-based marshalling introduced a lot of complexity to > > the > > > code and incorrect use of Kryo's numerous serializers caused some weird > > > hard-to-debug issues in the Ignite core (like duplicate cache entries > due > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go > > > figure!). Overall the Kryo-based solution is brittle and hard to > > maintain. > > > > > > I feel the correct solution to OSGi problem would be to > > > 1) enhance the OptimizedMarshaller to allow custom class resolution. > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the > > > original one) to be used in OSGi environment. > > > > > > Regards > > > Andrey > > > > > > > From: ra...@apache.org > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000 > > > > Subject: Re: OSGi migration may require a special marshaller > > > > To: dev@ignite.apache.org > > > > > > > > Hi Romain, > > > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your > > input. > > > > > > > > I'm thinking about the situation you describe and I wonder if you're > > > > exporting Ignite as an OSGi service which is then consumed from other > > > > bundles. Under this situation, it would be quite easy to reproduce > the > > > > behaviour you describe if Ignite is not resolving classes via the > TCCL. > > > > Need to dig deeper into that. > > > > > > > > Off the top of my head, there are two alternatives to solve it: > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already used) – > > we > > > > gotta be wary of possible regressions. > > > > 2. Create a special OSGi header Ignite-Export-Package so that bundles > > > > containing DTOs can expose packages to Ignite's marshallers. > > > > > > > > Regards, > > > > > > > > *Raúl Kripalani* > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data > > and > > > > Messaging Engineer > > > > http://about.me/raulkripalani | > > http://www.linkedin.com/in/raulkripalani > > > > http://blog.raulkr.net | twitter: @raulvk > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain < > > romain.gil...@misys.com> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'm really interested in this issue: > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some stuff > to > > > make > > > > > it work in our osgi environment. The main issue for us now is the > > > > > serialization. I think it you will have to rework the > > > OptimizedMarshaller > > > > > or any other marshaller that works with object that come from > outside > > > your > > > > > class space. > > > > > > > > > > We have try kryo that works. Kryo provide an extension point in > order > > > to > > > > > resolve the classes: > > > > > > > > > > > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java > > > > > . With this extension we are able to solve the problem of external > > > classes. > > > > > The only issue with kryo is that some classes need a certain care > in > > > the > > > > > serialization process and therefore a specialized serializer. > > > > > > > > > > So I would like to know from the community what do think of > changing > > > the > > > > > way the optimized marshaller works or introducing the support of > yet > > > > > another marshaller based on a kryo like technology? > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > Romain. > > > > > > > > > > > > > > > PS: I'm ready to help in the both cases. > > > > > "Misys" is the trade name of the Misys group of companies. This > email > > > and > > > > > any attachments have been scanned for known viruses using multiple > > > > > scanners. This email message is intended for the named recipient > > only. > > > It > > > > > may be privileged and/or confidential. If you are not the named > > > recipient > > > > > of this email please notify us immediately and do not copy it or > use > > > it for > > > > > any purpose, nor disclose its contents to any other person. This > > email > > > does > > > > > not constitute the commencement of legal relations between you and > > > Misys. > > > > > Please refer to the executed contract between you and the relevant > > > member > > > > > of the Misys group for the identity of the contracting party with > > > which you > > > > > are dealing. > > > > > > > > > > >
