Re: Missing ways to get access to Ignite CacheEntry

[Romain Gilles]

Hi Denis,
Thanks for you replay. And sorry to not double check it before. I see that
if I want to work with CacheEntry, I need to use the invoke method in order
to return the CacheEntry. This is the way I should do it. It sounds like
complicated for "just" getting an entry. But if you say this is the way I
will do it like that. I was just think that it could be a common use case
and therefore provide it as a shortcut.

Thanks,

Romain.

Le mer. 16 déc. 2015 à 11:34, Denis Magda <dma...@gridgain.com> a écrit :

> Hi Romain,
>
> As the current documentation of org.apache.ignite.cache.CacheEntry
> states it's possible to get a reference to CacheEntry and its version
> only for methods that are executed over local data set stored on a node.
> Among such methods are invoke & invokeAll and randomEntry.
>
> You probably can get a CacheEntry and its non null version when a cache
> iterator is in use but the version will be 'null', as far as I remember,
> for those entries that are loaded from remote nodes.
> Presently Ignite doesn't transfer the version from remote nodes as a
> part of response by perform reasons.
>
> Please elaborate more on your particular use case and what API you would
> like to add in order to support it.
>
> --
> Denis
>
> On 12/16/2015 12:58 PM, Romain Gilles wrote:
> > Hi Igniters,
> > I'm looking for a way to get access to the Ignite CacheEntry. For now
> this
> > is the ways I found:
> >
> > - Through the queris
> > - Through jsr 107 Cache Iterable
> > - Through jsr 107 Cache itterator
> > - Through IgniteCache::randomEntry()
> >
> > If I remember correctly it was possible to get the CacheEntry from a
> given
> > key in old version of gridgain community
> > version: GridCacheProjection::entry(K key) GridCacheEntry<K,V>
> > I think it could be a good to introduce this feature at IgniteCache
> level.
> > Or maybe there is an other way to do it.
> >
> > Thanks,
> >
> > Romain.
> >
>
>

Missing ways to get access to Ignite CacheEntry

[Romain Gilles]

Hi Igniters,
I'm looking for a way to get access to the Ignite CacheEntry. For now this
is the ways I found:

   - Through the queris
   - Through jsr 107 Cache Iterable
   - Through jsr 107 Cache itterator
   - Through IgniteCache::randomEntry()

If I remember correctly it was possible to get the CacheEntry from a given
key in old version of gridgain community
version: GridCacheProjection::entry(K key) GridCacheEntry
I think it could be a good to introduce this feature at IgniteCache level.
Or maybe there is an other way to do it.

Thanks,

Romain.

Re: 1.5 GA

[Romain Gilles]

Hi Igniters,
Yes but could please provide a push that includes the ignite osgi stuffs?
Regards,

Romain

Le lun. 14 déc. 2015 à 07:27, Dmitriy Setrakyan  a
écrit :

> Igniters,
>
> Since we have releases 1.5-b1, the community has reported several issues,
> which were addressed or are being addressed.
>
> Would it be realistic to plan releasing GA by the end of the week?
>
> D.
>

Re: OSGi integration ready for review

[Romain Gilles]

Hi Igniters,
I did the review. I term of impact the Raul commits change the manifest of
the overall produced jar files to introduce OSGi specific headers. This
will not impact original ignite usage as OSGi headers does are only for
OSGi environments. Some times maven-bundle-plugin can have side effect on
split package but I hope Raul has already checked it.
Then in term of code Raul has only added new project and code. He didn't
modify a single line of code in the already existing project therefore it
should not impact the existing code has it didn't touch it.
The only things I mention to Raul is to double check that the user class
loader (here Raul provide an OSGi aware user classloader) is used every
where in the serialization process:
 - Binary marshaller (new)
 - Optimized marshaller (old)
But here again it is just to make sure that in OSGi environment the new
feature provided by Raul will work without bad surprise it doesn't impact
the existing code.


Other than that it is a great feature and I'm excited to be able to use it!

Romain.

PS: some enhancement will be added in the future as discussed in the jira
ticket during the review. But it is an iterative approach and for now what
is provide is enough to start!

Le jeu. 3 déc. 2015 à 02:43, Dmitriy Setrakyan  a
écrit :

> Guys,
>
> Should we try to review and merge the OSGI integration before the final
> release? it does sound like it is additive and should not affect other
> functionality.
>
> My preference would be to get it in, so other OSGI-related integrations
> (e.g. Camel) would not have to wait for 1.6.
>
> Thoughts?
> D.
>
> On Mon, Nov 30, 2015 at 11:48 PM, Anton Vinogradov <
> avinogra...@gridgain.com
> > wrote:
>
> > Raul,
> >
> > I've added 1.6 space as a copy of 1.5.
> > But I gain no results at OSGi search.
> > Could you please specify what pages should be removed from 1.5 or revome
> > them by yourself?
> >
> > On Mon, Nov 30, 2015 at 6:18 PM, Yakov Zhdanov 
> > wrote:
> >
> > > Raul, as far as commits - they are mostly fixes for functionality, but
> do
> > > not add new features.
> > >
> > > Can someone create 1.6 space and move OSGi articles there? Perhaps,
> Anton
> > > V. can do it?
> > >
> > > --Yakov
> > >
> > > 2015-11-30 16:42 GMT+03:00 Raul Kripalani :
> > >
> > > > Dmitriy,
> > > >
> > > > I'm not going to judge if the communication of release timelines is
> > > > efficient. In fact, the release has still not been made and there's
> no
> > > > traffic in the ML about what's going on. Commits are still being
> > pushed,
> > > no
> > > > one knows anything. It seems communication is internal between
> > > committers.
> > > > But anyway, let's move ahead with OSGi.
> > > >
> > > > I have merged 1.5 into master, and master into ignite-1270. So it's
> now
> > > > mergeable into 1.6
> > > >
> > > > Someone needs to remove OSGi from the readme.io and push it into 1.6
> > > (not
> > > > me, I'm not happy with all this).
> > > >
> > > > Regards,
> > > >
> > > > *Raúl Kripalani*
> > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data
> > and
> > > > Messaging Engineer
> > > > http://about.me/raulkripalani |
> > http://www.linkedin.com/in/raulkripalani
> > > > http://blog.raulkr.net | twitter: @raulvk
> > > >
> > > > On Sat, Nov 28, 2015 at 5:13 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > On Fri, Nov 27, 2015 at 5:48 PM, Raul Kripalani 
> > > > wrote:
> > > > >
> > > > > > Aside from Dmitriy's questions, was there any blocker for someone
> > to
> > > > > review
> > > > > > the code? There has been no activity from committers.
> > > > > >
> > > > >
> > > > > As I mentioned last week, we were planning to cut a release this
> > > Friday,
> > > > > but I don’t think it happened. I too would like to get an update.
> > > Yakov,
> > > > > can you send out the status on open issues?
> > > > >
> > > > >
> > > > > > It was supposed to be merged for 1.5, and as far as I know, there
> > was
> > > > no
> > > > > > official announcement in the dev@ list that the release was
> being
> > > cut
> > > > > > today. How is the dev team supposed to be synchronised around
> these
> > > > > > milestones???
> > > > > >
> > > > >
> > > > > Raul, I am hoping the release will happen over the weekend,
> assuming
> > we
> > > > get
> > > > > an update. Even in that case, forcefully merging a new feature,
> that
> > is
> > > > not
> > > > > a critical bug fix, at this point would be too risky.
> > > > >
> > > > > My preference would be to cut another release with OSGI, Cammel,
> and
> > > > > Node.JS early January (no point of doing it sooner due to
> holidays).
> > > > Would
> > > > > do you think?
> > > > >
> > > > >
> > > > > >
> > > > > > To the release manager: please take out the OSGi documentation
> from
> > > 1.5
> > > > > in
> > > > > > readme.io and put it in 1.6.
> > > > > >
> > > > > > Regards,
> > > > > >
> > 

Re: Ignite-1.5 Release

[Romain Gilles]

Hi Raul and Igniters,
I have some comments for your pull request, one or two points. How does it
works do you plan to create a pull request? How can I comment if no,
through the mailing list?
Does this branch is planed to be merge for the 1.5?

Thanks in advance.

Romain.

Le mar. 24 nov. 2015 à 10:33, Dmitriy Setrakyan  a
écrit :

> Hi Raul,
>
> Thanks for taking on OSGI integration. I have reviewed the documentation
> and it looks good.
>
> At the risk of sounding pedantic, I think this link is best included as
> part of some other OSGI documentation page (no?):
> https://dash.readme.iro/project/apacheignite/v1.5/docs/introduction
> 
>
> Looking forward to seeing a final version.
>
> D.
>
> On Fri, Nov 20, 2015 at 10:47 PM, Raul Kripalani  wrote:
>
> > Igniters,
> >
> > OSGi integration is basically done. I need to give it another review
> pass.
> > I also want to see if I can OSGi-fy the new modules quickly (twitter,
> > flume, etc).
> >
> > Meanwhile I've worked on the documentation and it's pretty much done:
> >
> > *
> >
> >
> https://dash.readme.io/project/apacheignite/v1.5/docs/installation-in-apache-karaf
> > *
> >
> >
> https://dash.readme.io/project/apacheignite/v1.5/docs/installing-in-an-osgi-container
> > * https://dash.readme.io/project/apacheignite/v1.5/docs/introduction
> >
> > I expect to finish everything by Monday evening CET.
> >
> > Could someone please review it on Tuesday for merge into 1.5 next week?
> >
> > Branch is ignite-1270 in case you wanna peek.
> >
> > Thanks,
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Fri, Nov 20, 2015 at 4:15 PM, Yakov Zhdanov 
> > wrote:
> >
> > > Guys!
> > >
> > > Let's update status info.
> > >
> > > 1. Data grid performance optimizations.
> > >
> > > Most of the optimizations have been merged. However, there are several
> > > pending ones
> > >
> > >
> > > 2. Direct marshalling compactions: communication optimization -
> Valentin
> > >
> > > My understanding is that this will be merged in nearest hours.
> Valentin,
> > is
> > > that the case?
> > >
> > >
> > > 3. Introduce new binary format - Alexey Goncharuk and Vladimir Ozerov
> > >
> > > The work is mostly done and ignite-1282 will be merged to release
> branch
> > in
> > > several hours. Alexey, please let us know when you are done.
> > >
> > > 4. Data grid performance optimizations: SQL queries - Sergi and Semyon
> > >
> > > The problem here is that cache updates influence throughput of cache
> > > operations. As far as I know Semyon has investigated this and found
> > several
> > > places in code that could cause this. My understanding is that he will
> > > benchmark and merge this fix over the weekend.
> > >
> > >
> > > Thanks!
> > >
> > > --Yakov
> > >
> >
>

Re: Ignite-1.5 Release

[Romain Gilles]

Hi Raul,
Could you please give the pull request or the branch or the ignite ticket
number?
I will be please to have a look to your pull request.

Thanks,

Romain

Le jeu. 19 nov. 2015 à 04:01, Dmitriy Setrakyan  a
écrit :

> On Wed, Nov 18, 2015 at 2:17 PM, Raul Kripalani  wrote:
>
> > I merged the Camel Streamer today and I hope to finish the first phase of
> > OSGi tomorrow.
> >
> > I'll need someone to review the latter quickly if we want it merged for
> > 1.5... No code changes in existing codebase. Just a new osgi module and
> > many POM changes mainly to generate the manifests.
> >
>
> Raul, did you create a patch or a PR? I can’t find it anywhere.
>
>
> >
> > Should be a quick task.
> >
> > Regards,
> > Raúl.
> > On 18 Nov 2015 20:38, "Vladimir Ozerov"  wrote:
> >
> > > Hi Alex,
> > >
> > > Today I marged several big things into 1282 - reworked metadat and
> > > compacted footers optimization. I have on big ticket left -
> IGNITE-1917 -
> > > with marshalling microoptimizations. They should not conflict with
> > anything
> > > and I expect them to be finalized tomorrow. All optimizations are
> ready,
> > I
> > > just need to perform some cleanup and refactoring.
> > >
> > > On Wed, Nov 18, 2015 at 7:22 PM, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I merged performance optimizations for tx mode which gave up to 5%
> > > > improvement for tx-put benchmark to ignite-1.5.
> > > >
> > > > I also pushed finalizaing changes to Binary configuration in
> > ignite-1945
> > > > branch and currently waiting for TC. If all is ok, will cooperate
> with
> > > > Vladimir Ozerov and merge changes tomorrow.
> > > >
> > > > 2015-11-18 17:58 GMT+03:00 Semyon Boikov :
> > > >
> > > > > Hi,
> > > > >
> > > > > Yesterday I merged optimizations for tx update operations working
> > > single
> > > > > key, got ~7% improvement in tx-put benchmark.
> > > > >
> > > > > And today I finished to implement optimization for cache 'get'
> > > operation
> > > > > with single key. Got another benchmark results improvements: ~10%
> > with
> > > > > atomic-put-get, ~5% with tx-put-get. Need to verify TC and
> hopefully
> > > will
> > > > > merge these changes tomorrow.
> > > > >
> > > > > ​
> > > > >
> > > >
> > >
> >
>

Re: OSGi migration may require a special marshaller

[Romain Gilles]

Hi,
I have put my comments. Hope they will make the things progress :)
Should I start to implement this ticket or should I wait and see if Raoul
want to take it?

Romain

Le mar. 10 nov. 2015 à 02:58, Dmitriy Setrakyan <dsetrak...@apache.org> a
écrit :

> Thanks Raul, great points! I have created a ticket for the
> class-loader-detection design, described on wiki:
>
> https://issues.apache.org/jira/browse/IGNITE-1877
>
> D.
>
> On Mon, Nov 9, 2015 at 3:42 PM, Raul Kripalani <ra...@apache.org> wrote:
>
> > Hey Romain,
> >
> > I've updated the design proposal in the Wiki with my input. Could you
> > please have a look and let me know what you think?
> >
> > I'll implement a proof of concept of my proposed
> marshalling/unmarshalling
> > strategy and push it to Git so you can take it for a spin.
> >
> > Regards,
> >
> > *Raúl Kripalani*
> > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> > Messaging Engineer
> > http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> > http://blog.raulkr.net | twitter: @raulvk
> >
> > On Fri, Nov 6, 2015 at 4:05 PM, Romain Gilles <romain.gil...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I will put some sample code this WE. I'm exhausted sorry for the
> delay...
> > > Romain
> > >
> > > Le ven. 6 nov. 2015 à 00:45, Andrey Kornev <andrewkor...@hotmail.com>
> a
> > > écrit :
> > >
> > > > Hello,
> > > >
> > > > We've made an attempt to formalize the approach to Ignite's OSGi
> > > > enablement:
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility.
> > > > Please have a look and feel free to provide your positive feedback
> :)))
> > > >
> > > > Thanks
> > > > Andrey
> > > >
> > > > > From: dsetrak...@apache.org
> > > > > Date: Wed, 4 Nov 2015 09:26:17 -0800
> > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > To: dev@ignite.apache.org
> > > > >
> > > > > On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <
> > romain.gil...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Dmitriy,
> > > > > > I found this post that explain how to find a bundle based on its
> > > bundle
> > > > > > name and version.
> > > > > > This post explain for the past to now how to do it in the
> standard
> > > way
> > > > with
> > > > > > "pull" approach:
> > https://www.eclipse.org/forums/index.php/t/205719/
> > > > > > Regarding the version of OSGi you want to support then some
> > solutions
> > > > will
> > > > > > works some others will not.
> > > > > > There is an other way to do this stuff without using those "pull"
> > > style
> > > > > > approach based on BundleTracker. If you want I can give you the
> > code
> > > > to do
> > > > > > it with BundlTracker. I think with this solution you will
> support a
> > > > wider
> > > > > > range of OSGi version.
> > > > > >
> > > > >
> > > > > Romain, if you can provide a generic code sample to look up a
> > > ClassLoader
> > > > > in OSGI based on manifest properties, would be great.
> > > > >
> > > > >
> > > >
> > >
> >
>

Confluence comment / write restriction

[Romain Gilles]

Hi Igniters,
I would like to provide some feadbacks on the following ignite confluence
page: https://cwiki.apache.org/confluence/display/IGNITE/OSGI+Compatibility
Could you please, grant me the access right ?

Thanks in advance,

Romain Gilles.

Re: OSGi migration may require a special marshaller

[Romain Gilles]

Hi Dmitriy,
I found this post that explain how to find a bundle based on its bundle
name and version.
This post explain for the past to now how to do it in the standard way with
"pull" approach: https://www.eclipse.org/forums/index.php/t/205719/
Regarding the version of OSGi you want to support then some solutions will
works some others will not.
There is an other way to do this stuff without using those "pull" style
approach based on BundleTracker. If you want I can give you the code to do
it with BundlTracker. I think with this solution you will support a wider
range of OSGi version.

Romain


Le mer. 4 nov. 2015 à 04:07, Dmitriy Setrakyan <dsetrak...@apache.org> a
écrit :

> Andrey, et al,
>
> Can you provide a way on how to get a required Bundle object based on, say,
> bundle symbolic name and version?
>
> I have reached the end of the internet trying to find a way in OSGI to look
> up a Bundle based on something other than an actual Class belonging to that
> bundle, but no luck.
>
> D.
>
> On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <andrewkor...@hotmail.com>
> wrote:
>
> > Dmitriy,
> >
> > I think your approach will work, but I let Romain respond.
> >
> > Also, in terms of the implementation, please keep in mind that the
> > resolver must be called for each non-JDK and non-Ignite core class (it
> > would probably make sense to eschew storing class loaders for such
> classes
> > in favor of compactness of the serialized representation -- see below).
> > Also, it's worth keeping a cache of already resolved class loaders per
> > marshaller invocation (this is probably the context that Romain has
> > mentioned in his previous posting) to minimize the number of the resolver
> > calls.
> >
> > In terms of the resolver's implementation, the simplest way to serialize
> > the class loader would be by capturing two pieces of information (both
> > strings): the bundle symbolic name and the bundle version. This approach
> > however may result in bloating of the serialized representation: I'd
> > roughly estimate the overhead per element to be at least 20-30 bytes (the
> > length of the symbolic name string, plus the length of the version
> string).
> > There are way to reduce the overhead (like serializing the hash code of
> the
> > bundle id string rather than the string itself, and then come up with a
> > clever way of resolving the hash collisions), but they all come at cost
> of
> > increased complexity...
> >
> > An alternative approach would be rely on the special bundle id which is
> an
> > integer and is generated by the OSGi container. But in this case, all
> nodes
> > must ensure that all the bundles have consistent ids (bundle A with id 42
> > on node N1, has the same id on every other node) which is not easy --
> while
> > not entirely impossible -- to guarantee. As long as the nodes are
> > homogeneous (have the same set of bundles deployed) the OSGi container is
> > guaranteed to assign to the bundles the same ids.
> >
> > Thanks
> > Andrey
> >
> > > From: dsetrak...@apache.org
> > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Romain,
> > >
> > > In the upcoming release we will be deprecating the OptimizedMarshaller
> > and
> > > will be switching to a default internal marshaller (which is based on
> the
> > > new PortableMarshaller donated by GridGain).
> > >
> > > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > > instead of byte arrays. This will be pretty close to passing the
> stream,
> > as
> > > suggested by Andrey.
> > >
> > > Also, I still think that we should only resolve class loaders and not
> the
> > > class itself. The main reason is that Ignite will encode class names
> into
> > > an integer hash code and will store the integer->class-fqn mapping in
> > > internal replicated cache. I doubt users will get more efficient than
> an
> > > integer (4 bytes) for a class name.
> > >
> > > On the receiving side, once we are able to get the right class loader,
> we
> > > can easily get the proper class by calling
> > ClassLoader.findClass(class-fqn).
> > >
> > > Thoughts?
> > >
> > > D.
> > >
> > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <romain.gil...@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > > Maybe 

Re: OSGi migration may require a special marshaller

[Romain Gilles]

Hi Raul,
 - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
 - I like the point 2. Maybe for a graph of object that come from different
packages / bundles you may have to recursively capture the package version
of the individual element of your graph.
 - For the point 3 I wonder if it will not over-complexify the solution. As
a developer you will have to think about it. And it is not obvious in the
code where things are serialized or not. You may use lambda in your
application code therefore the current package become what you call a DTO
package. The current state of ignite does not enforce you to specify it for
"classical" classloading application. If you introduce this extra step for
OSGi ready application you will maybe discourage people to use OSGi.

To comeback to our solution. We start we a strong assumption: our cluster
is homogeneous in term of code so, of course, it simplify our life :).

BTW if you can introduce an extension point in the class resolution
mechanism it can be interesting for end user in order to allow them to
optimize it based on there specific use cases.

Romain.

Le mar. 3 nov. 2015 à 00:21, Raul Kripalani  a écrit :

> Hi Andrey,
>
> Thanks for the participation in this topic.
>
> I don't like the solution to incorporate the bundle symbolic name in the
> serialised form. Nothing guarantees that the classdef will be located under
> the same bundle in both source and target machines. We also have to take
> into account that OSGi allows for multiple versions of the same
> bundle/packages to co-exist in the same  container. So it becomes more
> complex.
>
> Using the TCCL should work when serialising, but it'll probably be of no
> use when deserialising on the other end.
>
> I need to enhance Ignite to:
>
> 1. Use the TCCL when marshalling on one end.
> 2. Incorporate the package version of the class in the serialised form if
> Ignite is running in an OSGi environment.
> 3. On the receiver end, discover cache entities / DTOs in all bundles
> through a custom OSGi manifest header or the like, as I explained before.
> Package version must be taken into account.
>
> What do you think?
>
> Raúl.
> On 2 Nov 2015 17:25, "Andrey Kornev"  wrote:
>
> > Raul,
> >
> > The fundamental hurdle we need to jump over to make Ignite OSGi-enabled
> is
> > the marshalling. More specifically the issue is with deserialization of
> the
> > classes that are provided by the bundles other than the Ignite bundle
> > itself.
> >
> > When the Ignite transport layer receives a message it needs to figure out
> > how to deserialize the bytes and for that it needs to know the bundle
> that
> > provides the class to be deserailized. At this point TCCL is of no use.
> To
> > make things more complex, the class may contain other classes that come
> > from other bundles, and so on recursively. This means that each object in
> > the hierarchy must be serialized with its bundle name (or bundle id), so
> > that the deserializer will then be able to correctly resolve the class
> > while traversing the object hierarchy during deserialization.
> >
> > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to
> plug
> > in a custom class resolver. Romain's solution was to use Kryo that does
> > provide a way to customize class resolution. It has solved the problem of
> > capturing the bundle info and he was able to successfully run Ignite as a
> > bundle in an OSGi container (after some repackaging and inclusion of the
> > manifest). But Kryo-based marshalling introduced a lot of complexity to
> the
> > code and incorrect use of Kryo's numerous serializers caused some weird
> > hard-to-debug issues in the Ignite core (like duplicate cache entries due
> > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go
> > figure!). Overall the Kryo-based solution is brittle and hard to
> maintain.
> >
> > I feel the correct solution to OSGi problem would be to
> > 1) enhance the OptimizedMarshaller to allow custom class resolution.
> > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the
> > original one) to be used in OSGi environment.
> >
> > Regards
> > Andrey
> >
> > > From: ra...@apache.org
> > > Date: Mon, 2 Nov 2015 12:41:47 +
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Hi Romain,
> > >
> > > I'm working on the OSGi compatibility of Ignite. I appreciate your
> input.
> > >
> > > I'm thinking about the situation you describe and I wonder if you're
> > > exporting Ignite as an OSGi service which is then consumed from other
> > > bundles. Under this situation, it would be quite easy to reproduce the
> > > behaviour you describe if Ignite is not resolving classes via the TCCL.
> > > Need to dig deeper into that.
> > >
> > > Off the top of my head, there are two alternatives to solve it:
> > >
> > > 1. Use the TCCL for marshalling/unmarshalling (if not already 

Re: OSGi migration may require a special marshaller

[Romain Gilles]

Hi Dmitriy,
I think your solution is good. Maybe I will change it a little bit... :P
I think you should delegate the Class resolution to the resolver. Because
for example with your current solution the marshaller may before (or after)
store the fqn of the class (maybe only the first time it encounter it) but
in order to save the classloader context resolution the implementation of
the resolver may have to save the package name of the given object and some
extra info therefore the content package name will be serialized two times.
Ok, it's not a big deal.
But now if the resolver identify that it can reuse the same class loader
for a couple of classes. It will be interesting for it to have a
serialization context in order to save, let say, classloader/id mapping in
order to save the id instead of a more longer representation.
So I propose something like that:
*public interface ClassResolver {*
*// This method is invoked on the sender side to *
*// marshal the information about the class.*
*// where the context is a map style object that is reset/new for each
object graph serialization.*
*public byte[] writeClass(Object o, Context context) throws
IgniteException;*

*// This method is invoked on the receiving side to*
*// retrieve the class based on provided information.*
*// where the context is a map style object that is reset/new for each
object graph serialization.*
*public Class readClass(byte[], Context context) throws
IgniteException;*
*}*

I think your proposal will solve our issue and maybe also open a door for
the osgi development.
Let me know what do you think about me proposal? :)

Thanks,

Romain

Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <dsetrak...@apache.org> a
écrit :

> Romain,
>
> Can you comment on the ClassLoaderResolver suggestion that I proposed
> earlier? Will it solve your problem?
>
> D.
>
> On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <romain.gil...@gmail.com>
> wrote:
>
> > Hi Raul,
> >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> >  - I like the point 2. Maybe for a graph of object that come from
> different
> > packages / bundles you may have to recursively capture the package
> version
> > of the individual element of your graph.
> >  - For the point 3 I wonder if it will not over-complexify the solution.
> As
> > a developer you will have to think about it. And it is not obvious in the
> > code where things are serialized or not. You may use lambda in your
> > application code therefore the current package become what you call a DTO
> > package. The current state of ignite does not enforce you to specify it
> for
> > "classical" classloading application. If you introduce this extra step
> for
> > OSGi ready application you will maybe discourage people to use OSGi.
> >
> > To comeback to our solution. We start we a strong assumption: our cluster
> > is homogeneous in term of code so, of course, it simplify our life :).
> >
> > BTW if you can introduce an extension point in the class resolution
> > mechanism it can be interesting for end user in order to allow them to
> > optimize it based on there specific use cases.
> >
> > Romain.
> >
> > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <r...@evosent.com> a écrit :
> >
> > > Hi Andrey,
> > >
> > > Thanks for the participation in this topic.
> > >
> > > I don't like the solution to incorporate the bundle symbolic name in
> the
> > > serialised form. Nothing guarantees that the classdef will be located
> > under
> > > the same bundle in both source and target machines. We also have to
> take
> > > into account that OSGi allows for multiple versions of the same
> > > bundle/packages to co-exist in the same  container. So it becomes more
> > > complex.
> > >
> > > Using the TCCL should work when serialising, but it'll probably be of
> no
> > > use when deserialising on the other end.
> > >
> > > I need to enhance Ignite to:
> > >
> > > 1. Use the TCCL when marshalling on one end.
> > > 2. Incorporate the package version of the class in the serialised form
> if
> > > Ignite is running in an OSGi environment.
> > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > through a custom OSGi manifest header or the like, as I explained
> before.
> > > Package version must be taken into account.
> > >
> > > What do you think?
> > >
> > > Raúl.
> > > On 2 Nov 2015 17:25, "Andrey Kornev" <andrewkor...@hotmail.com> wrote:
> > >
> > > > R

Re: OSGi migration may require a special marshaller

[Romain Gilles]

Hi Raul,
I had a look to what has been done in camel. I think here the use cases
came be different.
Let say I want to run a closure against the grid like this:

IgniteCompute compute  = ignite.compute();
// Execute closure on all cluster nodes.Collection res = compute.apply(
String::length,
Arrays.asList("How many characters".split(" "))
);
 // Add all the word lengths received from cluster nodes.

int total = res.stream().mapToInt(Integer::intValue).sum();

Then I think, even in more complex and realistic use cases, you may don't
want to export the implementation details of your closure. But this closure
will be serialized by the Ignite marshalling framework.

Romain.

Le mar. 3 nov. 2015 à 20:18, Raul Kripalani <ra...@apache.org> a écrit :

> Hey guys,
>
> About the TCCL, I need to dig deeper into how serialisation is being done
> now. If, at any point, we are resolving a class through the classloader of
> a particular class, e.g. Ignition.getClass().getClassLoader(), etc. we are
> using the class space of the bundle containing that class. If we are using
> a class from Ignite, we obviously wouldn't be able to find a custom DTO
> provided by a user.
>
> In Camel we have found this problem several times and we've solved it by
> changing the TCCL, or using the TCCL [1] [2] to resolve classes.
>
> As I said, I need to look into this deeper and I'll have some time on
> Thursday, so I hope you don't mind waiting a little bit. I have already
> "bundle-fied" most of Ignite modules in the corresponding branch [3], so my
> next step is to test out the Ignite examples inside an OSGi environment
> (Karaf 4).
>
> [1] https://issues.apache.org/jira/browse/CAMEL-5748
> [2] https://issues.apache.org/jira/browse/CAMEL-5722
> [3] https://github.com/apache/ignite/tree/ignite-1527
>
> Regards,
>
> *Raúl Kripalani*
> PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
> Messaging Engineer
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk
>
> On Tue, Nov 3, 2015 at 10:38 AM, Romain Gilles <romain.gil...@gmail.com>
> wrote:
>
> > Hi Raul,
> >  - Do you plan to use the TCCL when marshalling in OSGi env? If yes how?
> >  - I like the point 2. Maybe for a graph of object that come from
> different
> > packages / bundles you may have to recursively capture the package
> version
> > of the individual element of your graph.
> >  - For the point 3 I wonder if it will not over-complexify the solution.
> As
> > a developer you will have to think about it. And it is not obvious in the
> > code where things are serialized or not. You may use lambda in your
> > application code therefore the current package become what you call a DTO
> > package. The current state of ignite does not enforce you to specify it
> for
> > "classical" classloading application. If you introduce this extra step
> for
> > OSGi ready application you will maybe discourage people to use OSGi.
> >
> > To comeback to our solution. We start we a strong assumption: our cluster
> > is homogeneous in term of code so, of course, it simplify our life :).
> >
> > BTW if you can introduce an extension point in the class resolution
> > mechanism it can be interesting for end user in order to allow them to
> > optimize it based on there specific use cases.
> >
> > Romain.
> >
> > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <r...@evosent.com> a écrit :
> >
> > > Hi Andrey,
> > >
> > > Thanks for the participation in this topic.
> > >
> > > I don't like the solution to incorporate the bundle symbolic name in
> the
> > > serialised form. Nothing guarantees that the classdef will be located
> > under
> > > the same bundle in both source and target machines. We also have to
> take
> > > into account that OSGi allows for multiple versions of the same
> > > bundle/packages to co-exist in the same  container. So it becomes more
> > > complex.
> > >
> > > Using the TCCL should work when serialising, but it'll probably be of
> no
> > > use when deserialising on the other end.
> > >
> > > I need to enhance Ignite to:
> > >
> > > 1. Use the TCCL when marshalling on one end.
> > > 2. Incorporate the package version of the class in the serialised form
> if
> > > Ignite is running in an OSGi environment.
> > > 3. On the receiver end, discover cache entities / DTOs in all bundles
> > > through a custom OSGi manifest header or the like, as I explained
> before.
> &g