I do not think indexes is the right approach - set do not have indexes, and you will have to maintain additional counter for it in order to know when to stop.
>From what I see there are two distinct problems: 1) Broken recovery - this is just a bug which needs to be fixed. As soon as data is stored in real persistent cache, recovery of data structure state should be trivial task. 2) Heap items - this should not be a problem in common case when set contains moderate number of elements. If set is excessively large, then this is not the right structure for your use case and you should use standard IgniteCache API instead. What we can do is to optionally disable on-heap caching for specific set at the cost of lower performance if user wants so. On Wed, Feb 14, 2018 at 4:51 PM, Pavel Pereslegin <xxt...@gmail.com> wrote: > Hello, Igniters! > > I agree that solution with separate caches is not acceptable for a > large number of sets. > > So, I want to suggest one more way to implement IgniteSet that will > introduce element indexes (similar to IgniteQueue). To implement this > we can add head/tail indexes to IgniteSet header and for each > IgniteSet element store two key-value pairs: > setKey, index > index, setKey > > Indexes are required to support iterator and they should be continuous. > > Please, see detailed description in JIRA comment [1]. > > With such approach add/remove/contains operations will have O(1) time > complexity, iterator should work similar to current IgniteQueue > iterator, issues [2], [3] will be resolved, because PDS recovery will > work "out of the box" and we will not use JVM heap for duplicated > values. > > Btw, we can use this implementation only for collocated mode (map > keys/indexes to IgniteSet name) and use separate caches for > non-collocated mode. > > What do you think about this? > > [1] https://issues.apache.org/jira/browse/IGNITE-5553#comment-16364043 > [2] https://issues.apache.org/jira/browse/IGNITE-5553 > [3] https://issues.apache.org/jira/browse/IGNITE-7565 > > > 2018-02-13 9:33 GMT+03:00 Andrey Kuznetsov <stku...@gmail.com>: > > Indeed, all sets, regardless of whether they collocated or not, share > > single cache, and also use onheap data structures irresistable to > > checkpointing/recovery. > > > > 2018-02-13 2:14 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: > > > >> On Fri, Feb 9, 2018 at 6:26 PM, Andrey Kuznetsov <stku...@gmail.com> > >> wrote: > >> > >> > Hi all, > >> > > >> > Current set implementation has significant flaw: all set data are > >> > duplicated in onheap maps on _every_ node in order to make iterator() > and > >> > size(). For me it looks like simple yet ineffective implementation. > >> > Currently, these maps are damaged by checkpointing/recovery, and we > could > >> > patch them somehow. Another future change to Ignite caches can damage > >> them > >> > again. This looks fragile when datastructure is not entirely backed by > >> > caches. Pavel's proposal seems to be a reliable solution for > >> non-collocated > >> > sets. > >> > > >> > >> I would agree, but I was under an impression that non-collocated sets > are > >> already implemented this way. Am I wrong? > >> > > > > > > > > -- > > Best regards, > > Andrey Kuznetsov. >
