Hello, Igniters! I agree that solution with separate caches is not acceptable for a large number of sets.
So, I want to suggest one more way to implement IgniteSet that will introduce element indexes (similar to IgniteQueue). To implement this we can add head/tail indexes to IgniteSet header and for each IgniteSet element store two key-value pairs: setKey, index index, setKey Indexes are required to support iterator and they should be continuous. Please, see detailed description in JIRA comment [1]. With such approach add/remove/contains operations will have O(1) time complexity, iterator should work similar to current IgniteQueue iterator, issues [2], [3] will be resolved, because PDS recovery will work "out of the box" and we will not use JVM heap for duplicated values. Btw, we can use this implementation only for collocated mode (map keys/indexes to IgniteSet name) and use separate caches for non-collocated mode. What do you think about this? [1] https://issues.apache.org/jira/browse/IGNITE-5553#comment-16364043 [2] https://issues.apache.org/jira/browse/IGNITE-5553 [3] https://issues.apache.org/jira/browse/IGNITE-7565 2018-02-13 9:33 GMT+03:00 Andrey Kuznetsov <stku...@gmail.com>: > Indeed, all sets, regardless of whether they collocated or not, share > single cache, and also use onheap data structures irresistable to > checkpointing/recovery. > > 2018-02-13 2:14 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: > >> On Fri, Feb 9, 2018 at 6:26 PM, Andrey Kuznetsov <stku...@gmail.com> >> wrote: >> >> > Hi all, >> > >> > Current set implementation has significant flaw: all set data are >> > duplicated in onheap maps on _every_ node in order to make iterator() and >> > size(). For me it looks like simple yet ineffective implementation. >> > Currently, these maps are damaged by checkpointing/recovery, and we could >> > patch them somehow. Another future change to Ignite caches can damage >> them >> > again. This looks fragile when datastructure is not entirely backed by >> > caches. Pavel's proposal seems to be a reliable solution for >> non-collocated >> > sets. >> > >> >> I would agree, but I was under an impression that non-collocated sets are >> already implemented this way. Am I wrong? >> > > > > -- > Best regards, > Andrey Kuznetsov.
