I think the root issue is that we are trying to mix different cases in a single solution. What is the common usage patterns of sets? 1) Small mostly-read sets - current implementation is ideal for them - everything is available locally, on-heap and in deserialized form 2) Big data sets - IgniteCache API is the best candidate here; we can simply add a method "Set<K> IgniteCache.asSet()" which will return thin wrapper for Set interface around cache. An you will have recovery and heap-resistance with almost no additional efforts. There is no need to choose between one or another solution. We can support both.
As far as current igniteSet implementation - all issues described above will go away if you use approach suggested by Dmitry. I do not see a need for any major changes. Vladimir. On Mon, Feb 19, 2018 at 10:44 AM, Pavel Pereslegin <xxt...@gmail.com> wrote: > Hello Vladimir, > > > What we can do is to optionally disable on-heap caching for specific set > at the cost of lower performance if user wants so. > > I want to make sure that we are speaking about the same thing. By > "on-heap caching" I mean custom datastructure and not standard on-heap > cache, we can't disable it now, because it is the part of current > IgniteSet implementation. Is it acceptable to use GridOffHeapMap > optionally instead of ConcurrentHashMap for "off-heap mode"? > > > 2018-02-15 23:20 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: > > On Thu, Feb 15, 2018 at 6:08 AM, Vladimir Ozerov <voze...@gridgain.com> > > wrote: > > > >> I do not think indexes is the right approach - set do not have indexes, > and > >> you will have to maintain additional counter for it in order to know > when > >> to stop. > >> > >> From what I see there are two distinct problems: > >> 1) Broken recovery - this is just a bug which needs to be fixed. As > soon as > >> data is stored in real persistent cache, recovery of data structure > state > >> should be trivial task. > >> 2) Heap items - this should not be a problem in common case when set > >> contains moderate number of elements. If set is excessively large, then > >> this is not the right structure for your use case and you should use > >> standard IgniteCache API instead. What we can do is to optionally > disable > >> on-heap caching for specific set at the cost of lower performance if > user > >> wants so. > >> > > > > Vladimir, I am not sure I agree. In my view, set should be similar to > > cache, just a different API. I am not sure why we should make an > > assumptions that set data should be smaller than cache, especially given > > that it is a trivial task to implement a set based on Ignite cache API > (we > > could just store key-key mappings in cache instead of key-value mappings > > internally). > > > > Can you clarify why you believe that IgniteSet should need to have > on-heap > > entries? > > > > D. >
