I agree that solution with separate caches is not acceptable for a
large number of sets.
So, I want to suggest one more way to implement IgniteSet that will
introduce element indexes (similar to IgniteQueue). To implement this
we can add head/tail indexes to IgniteSet header and for each
IgniteSet element store two key-value pairs:
Indexes are required to support iterator and they should be continuous.
Please, see detailed description in JIRA comment .
With such approach add/remove/contains operations will have O(1) time
complexity, iterator should work similar to current IgniteQueue
iterator, issues ,  will be resolved, because PDS recovery will
work "out of the box" and we will not use JVM heap for duplicated
Btw, we can use this implementation only for collocated mode (map
keys/indexes to IgniteSet name) and use separate caches for
What do you think about this?
2018-02-13 9:33 GMT+03:00 Andrey Kuznetsov <stku...@gmail.com>:
> Indeed, all sets, regardless of whether they collocated or not, share
> single cache, and also use onheap data structures irresistable to
> 2018-02-13 2:14 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
>> On Fri, Feb 9, 2018 at 6:26 PM, Andrey Kuznetsov <stku...@gmail.com>
>> > Hi all,
>> > Current set implementation has significant flaw: all set data are
>> > duplicated in onheap maps on _every_ node in order to make iterator() and
>> > size(). For me it looks like simple yet ineffective implementation.
>> > Currently, these maps are damaged by checkpointing/recovery, and we could
>> > patch them somehow. Another future change to Ignite caches can damage
>> > again. This looks fragile when datastructure is not entirely backed by
>> > caches. Pavel's proposal seems to be a reliable solution for
>> > sets.
>> I would agree, but I was under an impression that non-collocated sets are
>> already implemented this way. Am I wrong?
> Best regards,
> Andrey Kuznetsov.