Re: [infinispan-dev] Starting caches in parallel
2011/8/4 Galder Zamarreño : > > On Aug 4, 2011, at 5:06 PM, Sanne Grinovero wrote: > >>> H, not sure about the warn message. Surely a no-op and if it's no-op >>> from the 2nd time onwards, what's the warning about? >> >> My reasoning is that if you're starting caches multiple times, then >> you're likely expecting Infinispan to support asymmetric clusters. > > Hmmm, I see what you mean but I think this needs further baking. You're > implying that if you do the following, "manik" cache could be started a > different time: > > cacheContainer.startCaches("galder", "sanne") -> starts galder and sanne > cacheContainer.startCaches("galder", "sanne") -> does nothing, why warn? > cacheContainer.startCaches("galder", "sanne", "manik") -> starts manik - yeah > could be seen as asymmetric clusters - maybe this one should show a warning > saying that "manik" was started at a different time to the other caches, and > point to the docu... +1 you could check for that, even better user experience. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On Aug 4, 2011, at 5:06 PM, Sanne Grinovero wrote: >> H, not sure about the warn message. Surely a no-op and if it's no-op >> from the 2nd time onwards, what's the warning about? > > My reasoning is that if you're starting caches multiple times, then > you're likely expecting Infinispan to support asymmetric clusters. Hmmm, I see what you mean but I think this needs further baking. You're implying that if you do the following, "manik" cache could be started a different time: cacheContainer.startCaches("galder", "sanne") -> starts galder and sanne cacheContainer.startCaches("galder", "sanne") -> does nothing, why warn? cacheContainer.startCaches("galder", "sanne", "manik") -> starts manik - yeah could be seen as asymmetric clusters - maybe this one should show a warning saying that "manik" was started at a different time to the other caches, and point to the docu... > > Many expect asymmetric clusters to just work, not all realize that > they are wrong until in a late development stage: it's obviously > unlikely that people test their application in cluster in its early > design phases. > This had bitten myself and it did no good to my project, and I still > think we're not making this limitation clear enough. A log is quite > likely going to be noticed, and I see no harm in it; users could save > a lot of time; I see no reason to not warn? > > Cheers, > Sanne > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
> H, not sure about the warn message. Surely a no-op and if it's no-op from > the 2nd time onwards, what's the warning about? My reasoning is that if you're starting caches multiple times, then you're likely expecting Infinispan to support asymmetric clusters. Many expect asymmetric clusters to just work, not all realize that they are wrong until in a late development stage: it's obviously unlikely that people test their application in cluster in its early design phases. This had bitten myself and it did no good to my project, and I still think we're not making this limitation clear enough. A log is quite likely going to be noticed, and I see no harm in it; users could save a lot of time; I see no reason to not warn? Cheers, Sanne ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On Aug 4, 2011, at 4:28 PM, Galder Zamarreño wrote: > > On Aug 4, 2011, at 4:22 PM, Sanne Grinovero wrote: > >> 2011/8/4 Galder Zamarreño : >>> >>> >> >> agreed, as discussed with Dan we won't throw exceptions but log a >> warning to recommend people use the startCaches(). >> >> Dan I still think the #startCaches method should at least log a >> warning if it's invoked more than once: it's very useful to be aware >> of the ISPN-658 as early as possible in the application design, and >> this is a good chance for us to detect the unsupported usage pattern. >> I think these two warnings combined can save a lot of time and stress, >> and won't affect any body who is doing it correctly already. > > H, not sure about the warn message. Surely a no-op and if it's no-op from > the 2nd time onwards, what's the warning about? Btw, whatever is done, start() and stop() need to be symmetric, and currently calling stop() on a already stopped cache does nothing. In fact, a debug message is shown to indicate that the 2nd call is ignored: AbstractComponentRegistry.stop(): getLog().debugf("Ignoring call to stop() as current state is %s", this); >> Cheers, >> Sanne >> >> ___ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On Aug 4, 2011, at 4:22 PM, Sanne Grinovero wrote: > 2011/8/4 Galder Zamarreño : >> >> On Aug 4, 2011, at 1:01 PM, Sanne Grinovero wrote: >> >>> 2011/8/4 Dan Berindei : >>> >>> Are you proposing a temporary API to make things work before ISPN-658 >>> is solved? I don't like the Future approach, it's still unclear that I >>> have to send all requests before blocking on any get. >>> I'd make a >>> >>> void startCaches(String... names); >>> >>> which implicitly includes the default cache too, and you throw an >>> exception if getCache() is used on an unstarted cache, and >>> unfortunately you should also throw an exception if startCaches() is >>> invoked more than once. >> >> We talk about this option too as indicated in >> https://gist.github.com/1124740, but I personally don't like it at this >> stage, since that'd be changing the programming model. People are expecting >> start caches with getCache(). It's been like that since 4.0 so what you're >> suggesting would be a bit change at this stage (5.0.0.CR8) IMO. Your >> suggestion would be worth considering for 6.x. > > agreed, as discussed with Dan we won't throw exceptions but log a > warning to recommend people use the startCaches(). > > Dan I still think the #startCaches method should at least log a > warning if it's invoked more than once: it's very useful to be aware > of the ISPN-658 as early as possible in the application design, and > this is a good chance for us to detect the unsupported usage pattern. > I think these two warnings combined can save a lot of time and stress, > and won't affect any body who is doing it correctly already. H, not sure about the warn message. Surely a no-op and if it's no-op from the 2nd time onwards, what's the warning about? > > +1 for Galder's fluent idea :) > > Cheers, > Sanne > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
2011/8/4 Galder Zamarreño : > > On Aug 4, 2011, at 1:01 PM, Sanne Grinovero wrote: > >> 2011/8/4 Dan Berindei : >>> >> >> Are you proposing a temporary API to make things work before ISPN-658 >> is solved? I don't like the Future approach, it's still unclear that I >> have to send all requests before blocking on any get. >> I'd make a >> >> void startCaches(String... names); >> >> which implicitly includes the default cache too, and you throw an >> exception if getCache() is used on an unstarted cache, and >> unfortunately you should also throw an exception if startCaches() is >> invoked more than once. > > We talk about this option too as indicated in > https://gist.github.com/1124740, but I personally don't like it at this > stage, since that'd be changing the programming model. People are expecting > start caches with getCache(). It's been like that since 4.0 so what you're > suggesting would be a bit change at this stage (5.0.0.CR8) IMO. Your > suggestion would be worth considering for 6.x. agreed, as discussed with Dan we won't throw exceptions but log a warning to recommend people use the startCaches(). Dan I still think the #startCaches method should at least log a warning if it's invoked more than once: it's very useful to be aware of the ISPN-658 as early as possible in the application design, and this is a good chance for us to detect the unsupported usage pattern. I think these two warnings combined can save a lot of time and stress, and won't affect any body who is doing it correctly already. +1 for Galder's fluent idea :) Cheers, Sanne ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On 4 Aug 2011, at 15:12, Galder Zamarreño wrote: > > On Aug 4, 2011, at 3:43 PM, Pete Muir wrote: > >> >> On 4 Aug 2011, at 14:01, Galder Zamarreño wrote: >> >>> >>> On Aug 4, 2011, at 1:01 PM, Sanne Grinovero wrote: >>> 2011/8/4 Dan Berindei : > Are you proposing a temporary API to make things work before ISPN-658 is solved? I don't like the Future approach, it's still unclear that I have to send all requests before blocking on any get. I'd make a void startCaches(String... names); which implicitly includes the default cache too, and you throw an exception if getCache() is used on an unstarted cache, and unfortunately you should also throw an exception if startCaches() is invoked more than once. >>> >>> We talk about this option too as indicated in >>> https://gist.github.com/1124740, but I personally don't like it at this >>> stage, since that'd be changing the programming model. People are expecting >>> start caches with getCache(). It's been like that since 4.0 so what you're >>> suggesting would be a bit change at this stage (5.0.0.CR8) IMO. Your >>> suggestion would be worth considering for 6.x. >> >> Dan's latest email contains an updated proposal, which would mean that >> getCache would still start the cache for 5.0 (and possibly 5.x) but would be >> "deprecated" (not recommended by docs, javadocs updated, warning logged) if >> called on an unstarted cache. startCaches would also be available in 5.x for >> those who do need to start multiple caches. >> >> WDYT? It seems to me this: >> >> * improves the API (makes the behavior of getCache more expected - just a >> getter, and completely idempotent) >> * indicates to the user that they do need to start the cache, and probably >> they will be more likely to start the cache at app start as well >> * doesn't break the existing API, but does offer a way to get around the >> problem > > Ok, let's go with this as long as the old method is left around. startCaches > would definitely make the API cleaner. > > Btw, let's make the API fluent, so instead of: Nice :-) > > void startCaches(String... names); > > Do: > > CacheContainer startCaches(String... names); > > To be able to do: > > startCaches(x).getCache(x) > > But we can't go final directly after making this change. We need at least > another CR to fix any potential issues that might require changing the API. > >> ___ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On Aug 4, 2011, at 3:43 PM, Pete Muir wrote: > > On 4 Aug 2011, at 14:01, Galder Zamarreño wrote: > >> >> On Aug 4, 2011, at 1:01 PM, Sanne Grinovero wrote: >> >>> 2011/8/4 Dan Berindei : >>> >>> Are you proposing a temporary API to make things work before ISPN-658 >>> is solved? I don't like the Future approach, it's still unclear that I >>> have to send all requests before blocking on any get. >>> I'd make a >>> >>> void startCaches(String... names); >>> >>> which implicitly includes the default cache too, and you throw an >>> exception if getCache() is used on an unstarted cache, and >>> unfortunately you should also throw an exception if startCaches() is >>> invoked more than once. >> >> We talk about this option too as indicated in >> https://gist.github.com/1124740, but I personally don't like it at this >> stage, since that'd be changing the programming model. People are expecting >> start caches with getCache(). It's been like that since 4.0 so what you're >> suggesting would be a bit change at this stage (5.0.0.CR8) IMO. Your >> suggestion would be worth considering for 6.x. > > Dan's latest email contains an updated proposal, which would mean that > getCache would still start the cache for 5.0 (and possibly 5.x) but would be > "deprecated" (not recommended by docs, javadocs updated, warning logged) if > called on an unstarted cache. startCaches would also be available in 5.x for > those who do need to start multiple caches. > > WDYT? It seems to me this: > > * improves the API (makes the behavior of getCache more expected - just a > getter, and completely idempotent) > * indicates to the user that they do need to start the cache, and probably > they will be more likely to start the cache at app start as well > * doesn't break the existing API, but does offer a way to get around the > problem Ok, let's go with this as long as the old method is left around. startCaches would definitely make the API cleaner. Btw, let's make the API fluent, so instead of: void startCaches(String... names); Do: CacheContainer startCaches(String... names); To be able to do: startCaches(x).getCache(x) But we can't go final directly after making this change. We need at least another CR to fix any potential issues that might require changing the API. > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On 4 Aug 2011, at 14:01, Galder Zamarreño wrote: > > On Aug 4, 2011, at 1:01 PM, Sanne Grinovero wrote: > >> 2011/8/4 Dan Berindei : >>> >> >> Are you proposing a temporary API to make things work before ISPN-658 >> is solved? I don't like the Future approach, it's still unclear that I >> have to send all requests before blocking on any get. >> I'd make a >> >> void startCaches(String... names); >> >> which implicitly includes the default cache too, and you throw an >> exception if getCache() is used on an unstarted cache, and >> unfortunately you should also throw an exception if startCaches() is >> invoked more than once. > > We talk about this option too as indicated in > https://gist.github.com/1124740, but I personally don't like it at this > stage, since that'd be changing the programming model. People are expecting > start caches with getCache(). It's been like that since 4.0 so what you're > suggesting would be a bit change at this stage (5.0.0.CR8) IMO. Your > suggestion would be worth considering for 6.x. Dan's latest email contains an updated proposal, which would mean that getCache would still start the cache for 5.0 (and possibly 5.x) but would be "deprecated" (not recommended by docs, javadocs updated, warning logged) if called on an unstarted cache. startCaches would also be available in 5.x for those who do need to start multiple caches. WDYT? It seems to me this: * improves the API (makes the behavior of getCache more expected - just a getter, and completely idempotent) * indicates to the user that they do need to start the cache, and probably they will be more likely to start the cache at app start as well * doesn't break the existing API, but does offer a way to get around the problem ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On Aug 4, 2011, at 1:01 PM, Sanne Grinovero wrote: > 2011/8/4 Dan Berindei : >> > > Are you proposing a temporary API to make things work before ISPN-658 > is solved? I don't like the Future approach, it's still unclear that I > have to send all requests before blocking on any get. > I'd make a > > void startCaches(String... names); > > which implicitly includes the default cache too, and you throw an > exception if getCache() is used on an unstarted cache, and > unfortunately you should also throw an exception if startCaches() is > invoked more than once. We talk about this option too as indicated in https://gist.github.com/1124740, but I personally don't like it at this stage, since that'd be changing the programming model. People are expecting start caches with getCache(). It's been like that since 4.0 so what you're suggesting would be a bit change at this stage (5.0.0.CR8) IMO. Your suggestion would be worth considering for 6.x. > > Cheers, > Sanne > ___ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On Thu, Aug 4, 2011 at 2:01 PM, Sanne Grinovero wrote: > 2011/8/4 Dan Berindei : >> Hi guys >> >> I've found a deadlock with transactions spanning multiple caches >> during rehashing if the joiner's caches are started sequentially (for >> more details see https://gist.github.com/1124740) >> After discussing a bit on IRC with Manik and Galderz it appears the >> only solution for 5.0.0.FINAL would be to have a mechanism to start >> all the caches in parallel. >> >> There are several options to implement that. All of them require the >> users to know that they should create the caches in parallel at >> application startup to avoid problems. >> >> 1. Advise the users about the problem, but let them create their own >> threads and call EmbeddedCacheManager.getCache() on those threads. >> >> >> 2. Add a method EmbeddedCacheManager.getCaches() to start multiple >> caches in parallel. >> >> This is not as straightforward as it may seem, first there is a >> question of whether to use template parameters or not: >> 2.a. Set getCaches(String... cacheNames); >> vs >> 2.b. Set> getCaches(String... cacheNames); >> It turns out the proper generic version should be Map> createCaches(String...), but even this version produces unchecked warnings. We'd also had to make the return type a map, otherwise the user has no way of getting a specific cache. >> I don't think having the same K and V for all the caches is going to >> be very common, so I'd go with 2.a. >> >> Then there is the problem of how to request the default cache. I think >> "" should be fine, but we'd need to document it and also change the >> regular getCache() methods to accept it. > > You could use org.infinispan.manager.CacheContainer.DEFAULT_CACHE_NAME. > I thought it was something hidden from the users, I wasn't aware it's in the public API. This is perfect. >> 3. Add an async version of EmbeddedCacheManager.getCache(): >> >> Future> getCacheAsync(String cacheName); >> Future> getCacheAsync(); >> >> This nicely sidesteps the generics issue in solution 2 and it's also >> easier for the users than solution 1, so it's currently my favourite. >> >> >> What do you think? > > I think it's very bad but it doesn't surprise me, since starting more > than a single cache in sequence brings you in the scenario of > ISPN-658, > unless you're quick enough. So yes it's not nice, but we highlighted > the problem since long I think? > The workaround for ISPN-658 is to start all caches on application startup. This scenario appeared specifically because I was trying to avoid an asymmetric cluster and moved the creation of all caches to a central location. > Are you proposing a temporary API to make things work before ISPN-658 > is solved? I don't like the Future approach, it's still unclear that I > have to send all requests before blocking on any get. The fix for ISPN-658 will most likely prevent existing nodes from sending request to a joiner until that joiner has finished the initial rehash, so I presume it will make my scenario impossible. There is another JIRA planned for 5.1 that is targeted more specifically as a workaround for ISPN-658: allow InboundInvocationHandler to create caches automatically when it has an incoming command for that cache (can't find the id now). > I'd make a > > void startCaches(String... names); > > which implicitly includes the default cache too, and you throw an > exception if getCache() is used on an unstarted cache, and > unfortunately you should also throw an exception if startCaches() is > invoked more than once. > I've had a long chat on IRC about this with Sanne and Pete, and our conclusion was that adding a startCaches method is the clearest option for the users. The intention of this API is only to start the caches, so the name should clearly state that. It will also eliminate an uncertainty for users: at the end of startCaches call they will know that the caches have started and that further getCache calls won't fail. We can't afford to break getCache() for users though (including ourselves), so getCache() will still start the cache. It should still print a warning when doing so, directing people to call startCaches instead. I don't think we should throw an exception if startCaches is invoked more than once either to match getCache() and all the other components. Cheers Dan ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
On 4 Aug 2011, at 12:01, Sanne Grinovero wrote: >> This is not as straightforward as it may seem, first there is a >> question of whether to use template parameters or not: >> 2.a. Set getCaches(String... cacheNames); >> vs >> 2.b. Set> getCaches(String... cacheNames); >> >> I don't think having the same K and V for all the caches is going to >> be very common, so I'd go with 2.a. Dan and I discussed this on IRC. The JLS says that using the raw type of a parameterized type is bad practice "The use of raw types is allowed only as a concession to compatibility of leg- acy code. The use of raw types in code written after the introduction of genericity into the Java programming language is strongly discouraged. It is possible that future versions of the Java programming language will disallow the use of raw types." The correct signature of 2a. is Set> which encodes that we never expect K,V to be same for all caches retrieved this way. If we do expect K, V to be the same in any case we should specify it as Set> as this allows the compiler to infer the generic type arguments, and doesn't prohibit Set>, or make any more typing for the user. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Starting caches in parallel
2011/8/4 Dan Berindei : > Hi guys > > I've found a deadlock with transactions spanning multiple caches > during rehashing if the joiner's caches are started sequentially (for > more details see https://gist.github.com/1124740) > After discussing a bit on IRC with Manik and Galderz it appears the > only solution for 5.0.0.FINAL would be to have a mechanism to start > all the caches in parallel. > > There are several options to implement that. All of them require the > users to know that they should create the caches in parallel at > application startup to avoid problems. > > 1. Advise the users about the problem, but let them create their own > threads and call EmbeddedCacheManager.getCache() on those threads. > > > 2. Add a method EmbeddedCacheManager.getCaches() to start multiple > caches in parallel. > > This is not as straightforward as it may seem, first there is a > question of whether to use template parameters or not: > 2.a. Set getCaches(String... cacheNames); > vs > 2.b. Set> getCaches(String... cacheNames); > > I don't think having the same K and V for all the caches is going to > be very common, so I'd go with 2.a. > > Then there is the problem of how to request the default cache. I think > "" should be fine, but we'd need to document it and also change the > regular getCache() methods to accept it. You could use org.infinispan.manager.CacheContainer.DEFAULT_CACHE_NAME. > 3. Add an async version of EmbeddedCacheManager.getCache(): > > Future> getCacheAsync(String cacheName); > Future> getCacheAsync(); > > This nicely sidesteps the generics issue in solution 2 and it's also > easier for the users than solution 1, so it's currently my favourite. > > > What do you think? I think it's very bad but it doesn't surprise me, since starting more than a single cache in sequence brings you in the scenario of ISPN-658, unless you're quick enough. So yes it's not nice, but we highlighted the problem since long I think? Are you proposing a temporary API to make things work before ISPN-658 is solved? I don't like the Future approach, it's still unclear that I have to send all requests before blocking on any get. I'd make a void startCaches(String... names); which implicitly includes the default cache too, and you throw an exception if getCache() is used on an unstarted cache, and unfortunately you should also throw an exception if startCaches() is invoked more than once. Cheers, Sanne ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
[infinispan-dev] Starting caches in parallel
Hi guys I've found a deadlock with transactions spanning multiple caches during rehashing if the joiner's caches are started sequentially (for more details see https://gist.github.com/1124740) After discussing a bit on IRC with Manik and Galderz it appears the only solution for 5.0.0.FINAL would be to have a mechanism to start all the caches in parallel. There are several options to implement that. All of them require the users to know that they should create the caches in parallel at application startup to avoid problems. 1. Advise the users about the problem, but let them create their own threads and call EmbeddedCacheManager.getCache() on those threads. 2. Add a method EmbeddedCacheManager.getCaches() to start multiple caches in parallel. This is not as straightforward as it may seem, first there is a question of whether to use template parameters or not: 2.a. Set getCaches(String... cacheNames); vs 2.b. Set> getCaches(String... cacheNames); I don't think having the same K and V for all the caches is going to be very common, so I'd go with 2.a. Then there is the problem of how to request the default cache. I think "" should be fine, but we'd need to document it and also change the regular getCache() methods to accept it. 3. Add an async version of EmbeddedCacheManager.getCache(): Future> getCacheAsync(String cacheName); Future> getCacheAsync(); This nicely sidesteps the generics issue in solution 2 and it's also easier for the users than solution 1, so it's currently my favourite. What do you think? Dan ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev