Re: Data eviction/expiration from Ignite persistence

[Dmitriy Setrakyan]

On Thu, Mar 15, 2018 at 12:31 PM, Denis Magda  wrote:

> >
> > Agree with AG. There is a difference between expiration and eviction. If
> an
> > entry is expired, then it should be removed from the store, regardless if
> > it is in memory or on disk.
>
>
> Well, then it works this way now depending on a memory configuration:
>
>- memory only mode: expired entry removed from memory storage
>- memory + Ignite persistence: expired entry removed from both memory
>and disk tiers
>- memory + 3rd party: expired entry is removed from the memory storage
>only.
>

I think it works correctly. Whenever Ignite is configured with a 3rd party
database, entries are usually expired from memory exactly because the
database may have a more recent value. This way, the more recent value will
be reloaded form database on next access.


> Let me know if I'm wrong somewhere. Otherwise, I'll improve the docs to
> bring more clarity.
>

I believe you are correct.


> > However, evicting from memory because there is not enough space does not
> remove an entry from the store.
>
>
> That was another topic of the discussion. How can we support the *eviction*
> from disk (by using a particular configuration parameter)?
>

I do not think we need to evict from disk. None of the other databases
support it, and I do not think it is a required feature. Also, it will be
hard to implement in my view.

Re: Data eviction/expiration from Ignite persistence

[Denis Magda]

>
> Agree with AG. There is a difference between expiration and eviction. If an
> entry is expired, then it should be removed from the store, regardless if
> it is in memory or on disk.


Well, then it works this way now depending on a memory configuration:

   - memory only mode: expired entry removed from memory storage
   - memory + Ignite persistence: expired entry removed from both memory
   and disk tiers
   - memory + 3rd party: expired entry is removed from the memory storage
   only.

Let me know if I'm wrong somewhere. Otherwise, I'll improve the docs to
bring more clarity.

However, evicting from memory because there is not enough space does not
> remove an entry from the store.


That was another topic of the discussion. How can we support the *eviction*
from disk (by using a particular configuration parameter)?

--
Denis

On Wed, Mar 14, 2018 at 7:01 PM, Dmitriy Setrakyan 
wrote:

> On Wed, Mar 14, 2018 at 1:41 PM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Denis,
> >
> > With the approach of Ignite Durable Memory there is no difference between
> > 'memory' and 'disk'. The data is expired from the Ignite data storage
> which
> > can be persisted or not. Before persistence was introduced, TTL was
> mostly
> > used when write-through was enabled, otherwise data was cleared from
> Ignite
> > data storage. Currently, the situation stays the same - if an entry is
> > expired, it is removed from the Ignite storage, which looks absolutely
> > consistent to me.
> >
>
> Agree with AG. There is a difference between expiration and eviction. If an
> entry is expired, then it should be removed from the store, regardless if
> it is in memory or on disk.
>
> However, evicting from memory because there is not enough space does not
> remove an entry from the store.
>
> D.
>

Re: Data eviction/expiration from Ignite persistence

[Dmitriy Setrakyan]

On Wed, Mar 14, 2018 at 1:41 PM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Denis,
>
> With the approach of Ignite Durable Memory there is no difference between
> 'memory' and 'disk'. The data is expired from the Ignite data storage which
> can be persisted or not. Before persistence was introduced, TTL was mostly
> used when write-through was enabled, otherwise data was cleared from Ignite
> data storage. Currently, the situation stays the same - if an entry is
> expired, it is removed from the Ignite storage, which looks absolutely
> consistent to me.
>

Agree with AG. There is a difference between expiration and eviction. If an
entry is expired, then it should be removed from the store, regardless if
it is in memory or on disk.

However, evicting from memory because there is not enough space does not
remove an entry from the store.

D.

Re: Data eviction/expiration from Ignite persistence

[Alexey Goncharuk]

Denis,

With the approach of Ignite Durable Memory there is no difference between
'memory' and 'disk'. The data is expired from the Ignite data storage which
can be persisted or not. Before persistence was introduced, TTL was mostly
used when write-through was enabled, otherwise data was cleared from Ignite
data storage. Currently, the situation stays the same - if an entry is
expired, it is removed from the Ignite storage, which looks absolutely
consistent to me.

2018-03-13 21:30 GMT+03:00 Denis Magda :

> Alexey,
>
> My understanding was that the expiration policies worked for data in RAM
> only. Ok, if an expired entry is removed from both RAM and Ignite
> persistence then what happens if a cache store is used instead of Ignite
> storage? Do we remove expired entries from RDBMs, Cassandra, etc? My guess
> that we don't which doesn't look consistent product wide.
>
> --
> Denis
>
> On Tue, Mar 13, 2018 at 12:50 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Denis,
> >
> > What do you mean by 'current behavior when data is evicted from the
> memory
> > only'? TTL expiration effectively means that the corresponding key-value
> > pairs are destroyed. If you are talking about page replacement, then
> there
> > is no way to do this on per-key basis because a page must be replaced as
> a
> > whole and it makes no sense to track keys.
> >
> > --AG
> >
> > 2018-03-13 0:03 GMT+03:00 Denis Magda :
> >
> > > Dmitriy,
> > >
> > > It will break the current default behavior when data is evicted from
> the
> > > memory only, and I would disagree that it's a right decision overall.
> > >
> > > There are many scenarios when users need to have the eviction in the
> > memory
> > > layer and preserve data on disk for later usage. So, can we keep what
> we
> > > have now and merely expand the eviction to disk if the user requires
> it?
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov 
> > > wrote:
> > >
> > > > Denis,
> > > >
> > > > I suppose there is no configuration will be required. If TTL
> configured
> > > > entry will be removed from disk & memory both.
> > > >
> > > > SIncerely,
> > > > Dmitriy Pavlov
> > > >
> > > > пн, 12 мар. 2018 г. в 23:32, Denis Magda :
> > > >
> > > > > Alexey, Dmitriy,
> > > > >
> > > > > What would be the configuration parameter that defines if the
> > eviction
> > > > > should happen in the in-memory layer only or in both memory and
> disk?
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com> wrote:
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > Yes, the entries will be removed from both memory and
> persistence.
> > > > > >
> > > > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com>:
> > > > > >
> > > > > > > Alex,
> > > > > > >
> > > > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> > > expired
> > > > > > entry
> > > > > > > be removed from both memory and persistence?
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > > > alexey.goncha...@gmail.com> wrote:
> > > > > > >
> > > > > > > > The ticket [1] is in patch available state looks good, the
> only
> > > > thing
> > > > > > > left
> > > > > > > > there is to transfer old entries to new storage. I hope
> Andrey
> > > will
> > > > > > have
> > > > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > > > >
> > > > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> > > > > > > >
> > > > > > > > > Val,
> > > > > > > > >
> > > > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please
> > chime
> > > > in.
> > > > > > > > >
> > > > > > > > > In general, the more deployments the persistence will get
> the
> > > > more
> > > > > > > demand
> > > > > > > > > we will see for that capability. Personally, I'd create a
> > > ticket
> > > > > for
> > > > > > > now
> > > > > > > > > and put it off to our backlog.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Denis
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > > > dsetrak...@apache.org
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > > > dpavlov@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > As far as I know there is no plans. Denis please
> correct
> > me
> > > > if
> > > > > > I'm
> > > > > > > > > wrong.
> > > > > > > > > > >
> > > > > > > > > > > But users found these or similar bugs, it seems we need
> > to
> > > > > > support
> > > > > > > > PDS
> > > > > > > > > 

Re: Data eviction/expiration from Ignite persistence

[Denis Magda]

Alexey,

My understanding was that the expiration policies worked for data in RAM
only. Ok, if an expired entry is removed from both RAM and Ignite
persistence then what happens if a cache store is used instead of Ignite
storage? Do we remove expired entries from RDBMs, Cassandra, etc? My guess
that we don't which doesn't look consistent product wide.

--
Denis

On Tue, Mar 13, 2018 at 12:50 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Denis,
>
> What do you mean by 'current behavior when data is evicted from the memory
> only'? TTL expiration effectively means that the corresponding key-value
> pairs are destroyed. If you are talking about page replacement, then there
> is no way to do this on per-key basis because a page must be replaced as a
> whole and it makes no sense to track keys.
>
> --AG
>
> 2018-03-13 0:03 GMT+03:00 Denis Magda :
>
> > Dmitriy,
> >
> > It will break the current default behavior when data is evicted from the
> > memory only, and I would disagree that it's a right decision overall.
> >
> > There are many scenarios when users need to have the eviction in the
> memory
> > layer and preserve data on disk for later usage. So, can we keep what we
> > have now and merely expand the eviction to disk if the user requires it?
> >
> > --
> > Denis
> >
> >
> > On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov 
> > wrote:
> >
> > > Denis,
> > >
> > > I suppose there is no configuration will be required. If TTL configured
> > > entry will be removed from disk & memory both.
> > >
> > > SIncerely,
> > > Dmitriy Pavlov
> > >
> > > пн, 12 мар. 2018 г. в 23:32, Denis Magda :
> > >
> > > > Alexey, Dmitriy,
> > > >
> > > > What would be the configuration parameter that defines if the
> eviction
> > > > should happen in the in-memory layer only or in both memory and disk?
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com> wrote:
> > > >
> > > > > Val,
> > > > >
> > > > > Yes, the entries will be removed from both memory and persistence.
> > > > >
> > > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com>:
> > > > >
> > > > > > Alex,
> > > > > >
> > > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> > expired
> > > > > entry
> > > > > > be removed from both memory and persistence?
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > > alexey.goncha...@gmail.com> wrote:
> > > > > >
> > > > > > > The ticket [1] is in patch available state looks good, the only
> > > thing
> > > > > > left
> > > > > > > there is to transfer old entries to new storage. I hope Andrey
> > will
> > > > > have
> > > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > > >
> > > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> > > > > > >
> > > > > > > > Val,
> > > > > > > >
> > > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please
> chime
> > > in.
> > > > > > > >
> > > > > > > > In general, the more deployments the persistence will get the
> > > more
> > > > > > demand
> > > > > > > > we will see for that capability. Personally, I'd create a
> > ticket
> > > > for
> > > > > > now
> > > > > > > > and put it off to our backlog.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Denis
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > > dpavlov@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > As far as I know there is no plans. Denis please correct
> me
> > > if
> > > > > I'm
> > > > > > > > wrong.
> > > > > > > > > >
> > > > > > > > > > But users found these or similar bugs, it seems we need
> to
> > > > > support
> > > > > > > PDS
> > > > > > > > > and
> > > > > > > > > > TTL.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > We should either support it or throw a clear  exception on
> > > > startup
> > > > > > > > clearly
> > > > > > > > > stating that it is not. I suppose, it should be the latter
> > for
> > > > now.
> > > > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Alexey Goncharuk]

Dmitriy,

The TTL map is a regular B+Tree-based map with the key being a pair (expire
time + key). It is obviously stored in memory when there is no persistence.
When persistence is enabled, it is handled as any other index tree - the
page replacement algorithm is applied. No heap is consumed for this tree.

--AG

2018-03-13 15:30 GMT+03:00 Dmitriy Setrakyan :

>
>
> On Tue, Mar 13, 2018 at 3:50 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
>> Denis,
>>
>> What do you mean by 'current behavior when data is evicted from the memory
>> only'? TTL expiration effectively means that the corresponding key-value
>> pairs are destroyed.
>
>
> AG, can you please explain how this is implemented? Are we really keeping
> the whole TTL map in memory?
>
>

Re: Data eviction/expiration from Ignite persistence

[Dmitriy Setrakyan]

On Tue, Mar 13, 2018 at 3:50 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Denis,
>
> What do you mean by 'current behavior when data is evicted from the memory
> only'? TTL expiration effectively means that the corresponding key-value
> pairs are destroyed.


AG, can you please explain how this is implemented? Are we really keeping
the whole TTL map in memory?

Re: Data eviction/expiration from Ignite persistence

[Dmitry Pavlov]

Hi Denis,

I've created summary of terms here
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Eviction,rotationandexpiration

Hope this should solve a lot of miscommunications.

In this thread I meant only Expiration (aka TTL)

Sincerely,
Dmitriy Pavlov

вт, 13 мар. 2018 г. в 10:50, Alexey Goncharuk :

> Denis,
>
> What do you mean by 'current behavior when data is evicted from the memory
> only'? TTL expiration effectively means that the corresponding key-value
> pairs are destroyed. If you are talking about page replacement, then there
> is no way to do this on per-key basis because a page must be replaced as a
> whole and it makes no sense to track keys.
>
> --AG
>
> 2018-03-13 0:03 GMT+03:00 Denis Magda :
>
> > Dmitriy,
> >
> > It will break the current default behavior when data is evicted from the
> > memory only, and I would disagree that it's a right decision overall.
> >
> > There are many scenarios when users need to have the eviction in the
> memory
> > layer and preserve data on disk for later usage. So, can we keep what we
> > have now and merely expand the eviction to disk if the user requires it?
> >
> > --
> > Denis
> >
> >
> > On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov 
> > wrote:
> >
> > > Denis,
> > >
> > > I suppose there is no configuration will be required. If TTL configured
> > > entry will be removed from disk & memory both.
> > >
> > > SIncerely,
> > > Dmitriy Pavlov
> > >
> > > пн, 12 мар. 2018 г. в 23:32, Denis Magda :
> > >
> > > > Alexey, Dmitriy,
> > > >
> > > > What would be the configuration parameter that defines if the
> eviction
> > > > should happen in the in-memory layer only or in both memory and disk?
> > > >
> > > > --
> > > > Denis
> > > >
> > > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com> wrote:
> > > >
> > > > > Val,
> > > > >
> > > > > Yes, the entries will be removed from both memory and persistence.
> > > > >
> > > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com>:
> > > > >
> > > > > > Alex,
> > > > > >
> > > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> > expired
> > > > > entry
> > > > > > be removed from both memory and persistence?
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > > alexey.goncha...@gmail.com> wrote:
> > > > > >
> > > > > > > The ticket [1] is in patch available state looks good, the only
> > > thing
> > > > > > left
> > > > > > > there is to transfer old entries to new storage. I hope Andrey
> > will
> > > > > have
> > > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > > >
> > > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> > > > > > >
> > > > > > > > Val,
> > > > > > > >
> > > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please
> chime
> > > in.
> > > > > > > >
> > > > > > > > In general, the more deployments the persistence will get the
> > > more
> > > > > > demand
> > > > > > > > we will see for that capability. Personally, I'd create a
> > ticket
> > > > for
> > > > > > now
> > > > > > > > and put it off to our backlog.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Denis
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > > dpavlov@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > As far as I know there is no plans. Denis please correct
> me
> > > if
> > > > > I'm
> > > > > > > > wrong.
> > > > > > > > > >
> > > > > > > > > > But users found these or similar bugs, it seems we need
> to
> > > > > support
> > > > > > > PDS
> > > > > > > > > and
> > > > > > > > > > TTL.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > We should either support it or throw a clear  exception on
> > > > startup
> > > > > > > > clearly
> > > > > > > > > stating that it is not. I suppose, it should be the latter
> > for
> > > > now.
> > > > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Alexey Goncharuk]

Denis,

What do you mean by 'current behavior when data is evicted from the memory
only'? TTL expiration effectively means that the corresponding key-value
pairs are destroyed. If you are talking about page replacement, then there
is no way to do this on per-key basis because a page must be replaced as a
whole and it makes no sense to track keys.

--AG

2018-03-13 0:03 GMT+03:00 Denis Magda :

> Dmitriy,
>
> It will break the current default behavior when data is evicted from the
> memory only, and I would disagree that it's a right decision overall.
>
> There are many scenarios when users need to have the eviction in the memory
> layer and preserve data on disk for later usage. So, can we keep what we
> have now and merely expand the eviction to disk if the user requires it?
>
> --
> Denis
>
>
> On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov 
> wrote:
>
> > Denis,
> >
> > I suppose there is no configuration will be required. If TTL configured
> > entry will be removed from disk & memory both.
> >
> > SIncerely,
> > Dmitriy Pavlov
> >
> > пн, 12 мар. 2018 г. в 23:32, Denis Magda :
> >
> > > Alexey, Dmitriy,
> > >
> > > What would be the configuration parameter that defines if the eviction
> > > should happen in the in-memory layer only or in both memory and disk?
> > >
> > > --
> > > Denis
> > >
> > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com> wrote:
> > >
> > > > Val,
> > > >
> > > > Yes, the entries will be removed from both memory and persistence.
> > > >
> > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com>:
> > > >
> > > > > Alex,
> > > > >
> > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> expired
> > > > entry
> > > > > be removed from both memory and persistence?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com> wrote:
> > > > >
> > > > > > The ticket [1] is in patch available state looks good, the only
> > thing
> > > > > left
> > > > > > there is to transfer old entries to new storage. I hope Andrey
> will
> > > > have
> > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > >
> > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> > > > > >
> > > > > > > Val,
> > > > > > >
> > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please chime
> > in.
> > > > > > >
> > > > > > > In general, the more deployments the persistence will get the
> > more
> > > > > demand
> > > > > > > we will see for that capability. Personally, I'd create a
> ticket
> > > for
> > > > > now
> > > > > > > and put it off to our backlog.
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > dpavlov@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > As far as I know there is no plans. Denis please correct me
> > if
> > > > I'm
> > > > > > > wrong.
> > > > > > > > >
> > > > > > > > > But users found these or similar bugs, it seems we need to
> > > > support
> > > > > > PDS
> > > > > > > > and
> > > > > > > > > TTL.
> > > > > > > > >
> > > > > > > >
> > > > > > > > We should either support it or throw a clear  exception on
> > > startup
> > > > > > > clearly
> > > > > > > > stating that it is not. I suppose, it should be the latter
> for
> > > now.
> > > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Dmitriy Setrakyan]

To be honest, I am not clear how this would be implemented. Currently, we
keep TTL map in memory and it fits in memory. However, if we start tracking
entries on disk, the TTL map will grow too large and may not fit in memory
any longer.

Can someone explain how this will be handled?

D.

On Mon, Mar 12, 2018 at 2:03 PM, Denis Magda  wrote:

> Dmitriy,
>
> It will break the current default behavior when data is evicted from the
> memory only, and I would disagree that it's a right decision overall.
>
> There are many scenarios when users need to have the eviction in the memory
> layer and preserve data on disk for later usage. So, can we keep what we
> have now and merely expand the eviction to disk if the user requires it?
>
> --
> Denis
>
>
> On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov 
> wrote:
>
> > Denis,
> >
> > I suppose there is no configuration will be required. If TTL configured
> > entry will be removed from disk & memory both.
> >
> > SIncerely,
> > Dmitriy Pavlov
> >
> > пн, 12 мар. 2018 г. в 23:32, Denis Magda :
> >
> > > Alexey, Dmitriy,
> > >
> > > What would be the configuration parameter that defines if the eviction
> > > should happen in the in-memory layer only or in both memory and disk?
> > >
> > > --
> > > Denis
> > >
> > > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com> wrote:
> > >
> > > > Val,
> > > >
> > > > Yes, the entries will be removed from both memory and persistence.
> > > >
> > > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com>:
> > > >
> > > > > Alex,
> > > > >
> > > > > What is behavior going to be after IGNITE-5874 is fixed? Will
> expired
> > > > entry
> > > > > be removed from both memory and persistence?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com> wrote:
> > > > >
> > > > > > The ticket [1] is in patch available state looks good, the only
> > thing
> > > > > left
> > > > > > there is to transfer old entries to new storage. I hope Andrey
> will
> > > > have
> > > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > > >
> > > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> > > > > >
> > > > > > > Val,
> > > > > > >
> > > > > > > I'd like to hear Alexey G. opinion on this? Alex, please chime
> > in.
> > > > > > >
> > > > > > > In general, the more deployments the persistence will get the
> > more
> > > > > demand
> > > > > > > we will see for that capability. Personally, I'd create a
> ticket
> > > for
> > > > > now
> > > > > > > and put it off to our backlog.
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > > dpavlov@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > As far as I know there is no plans. Denis please correct me
> > if
> > > > I'm
> > > > > > > wrong.
> > > > > > > > >
> > > > > > > > > But users found these or similar bugs, it seems we need to
> > > > support
> > > > > > PDS
> > > > > > > > and
> > > > > > > > > TTL.
> > > > > > > > >
> > > > > > > >
> > > > > > > > We should either support it or throw a clear  exception on
> > > startup
> > > > > > > clearly
> > > > > > > > stating that it is not. I suppose, it should be the latter
> for
> > > now.
> > > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Denis Magda]

Dmitriy,

It will break the current default behavior when data is evicted from the
memory only, and I would disagree that it's a right decision overall.

There are many scenarios when users need to have the eviction in the memory
layer and preserve data on disk for later usage. So, can we keep what we
have now and merely expand the eviction to disk if the user requires it?

--
Denis


On Mon, Mar 12, 2018 at 1:35 PM, Dmitry Pavlov 
wrote:

> Denis,
>
> I suppose there is no configuration will be required. If TTL configured
> entry will be removed from disk & memory both.
>
> SIncerely,
> Dmitriy Pavlov
>
> пн, 12 мар. 2018 г. в 23:32, Denis Magda :
>
> > Alexey, Dmitriy,
> >
> > What would be the configuration parameter that defines if the eviction
> > should happen in the in-memory layer only or in both memory and disk?
> >
> > --
> > Denis
> >
> > On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> > alexey.goncha...@gmail.com> wrote:
> >
> > > Val,
> > >
> > > Yes, the entries will be removed from both memory and persistence.
> > >
> > > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > >
> > > > Alex,
> > > >
> > > > What is behavior going to be after IGNITE-5874 is fixed? Will expired
> > > entry
> > > > be removed from both memory and persistence?
> > > >
> > > > -Val
> > > >
> > > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com> wrote:
> > > >
> > > > > The ticket [1] is in patch available state looks good, the only
> thing
> > > > left
> > > > > there is to transfer old entries to new storage. I hope Andrey will
> > > have
> > > > > time to finish this soon, so we can target the fix for 2.5.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > > >
> > > > > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > I'd like to hear Alexey G. opinion on this? Alex, please chime
> in.
> > > > > >
> > > > > > In general, the more deployments the persistence will get the
> more
> > > > demand
> > > > > > we will see for that capability. Personally, I'd create a ticket
> > for
> > > > now
> > > > > > and put it off to our backlog.
> > > > > >
> > > > > > --
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > > dpavlov@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > As far as I know there is no plans. Denis please correct me
> if
> > > I'm
> > > > > > wrong.
> > > > > > > >
> > > > > > > > But users found these or similar bugs, it seems we need to
> > > support
> > > > > PDS
> > > > > > > and
> > > > > > > > TTL.
> > > > > > > >
> > > > > > >
> > > > > > > We should either support it or throw a clear  exception on
> > startup
> > > > > > clearly
> > > > > > > stating that it is not. I suppose, it should be the latter for
> > now.
> > > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Dmitry Pavlov]

Denis,

I suppose there is no configuration will be required. If TTL configured
entry will be removed from disk & memory both.

SIncerely,
Dmitriy Pavlov

пн, 12 мар. 2018 г. в 23:32, Denis Magda :

> Alexey, Dmitriy,
>
> What would be the configuration parameter that defines if the eviction
> should happen in the in-memory layer only or in both memory and disk?
>
> --
> Denis
>
> On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Val,
> >
> > Yes, the entries will be removed from both memory and persistence.
> >
> > 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> > > Alex,
> > >
> > > What is behavior going to be after IGNITE-5874 is fixed? Will expired
> > entry
> > > be removed from both memory and persistence?
> > >
> > > -Val
> > >
> > > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com> wrote:
> > >
> > > > The ticket [1] is in patch available state looks good, the only thing
> > > left
> > > > there is to transfer old entries to new storage. I hope Andrey will
> > have
> > > > time to finish this soon, so we can target the fix for 2.5.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > > >
> > > > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> > > >
> > > > > Val,
> > > > >
> > > > > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> > > > >
> > > > > In general, the more deployments the persistence will get the more
> > > demand
> > > > > we will see for that capability. Personally, I'd create a ticket
> for
> > > now
> > > > > and put it off to our backlog.
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > >
> > > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> > dpavlov@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > As far as I know there is no plans. Denis please correct me if
> > I'm
> > > > > wrong.
> > > > > > >
> > > > > > > But users found these or similar bugs, it seems we need to
> > support
> > > > PDS
> > > > > > and
> > > > > > > TTL.
> > > > > > >
> > > > > >
> > > > > > We should either support it or throw a clear  exception on
> startup
> > > > > clearly
> > > > > > stating that it is not. I suppose, it should be the latter for
> now.
> > > > > > Dmitriy, any chance you can file a ticket for it?
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Denis Magda]

Alexey, Dmitriy,

What would be the configuration parameter that defines if the eviction
should happen in the in-memory layer only or in both memory and disk?

--
Denis

On Mon, Mar 12, 2018 at 9:22 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Val,
>
> Yes, the entries will be removed from both memory and persistence.
>
> 2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Alex,
> >
> > What is behavior going to be after IGNITE-5874 is fixed? Will expired
> entry
> > be removed from both memory and persistence?
> >
> > -Val
> >
> > On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> > alexey.goncha...@gmail.com> wrote:
> >
> > > The ticket [1] is in patch available state looks good, the only thing
> > left
> > > there is to transfer old entries to new storage. I hope Andrey will
> have
> > > time to finish this soon, so we can target the fix for 2.5.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> > >
> > > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> > >
> > > > Val,
> > > >
> > > > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> > > >
> > > > In general, the more deployments the persistence will get the more
> > demand
> > > > we will see for that capability. Personally, I'd create a ticket for
> > now
> > > > and put it off to our backlog.
> > > >
> > > > --
> > > > Denis
> > > >
> > > >
> > > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov <
> dpavlov@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > As far as I know there is no plans. Denis please correct me if
> I'm
> > > > wrong.
> > > > > >
> > > > > > But users found these or similar bugs, it seems we need to
> support
> > > PDS
> > > > > and
> > > > > > TTL.
> > > > > >
> > > > >
> > > > > We should either support it or throw a clear  exception on startup
> > > > clearly
> > > > > stating that it is not. I suppose, it should be the latter for now.
> > > > > Dmitriy, any chance you can file a ticket for it?
> > > > >
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Alexey Goncharuk]

Val,

Yes, the entries will be removed from both memory and persistence.

2018-03-12 19:20 GMT+03:00 Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Alex,
>
> What is behavior going to be after IGNITE-5874 is fixed? Will expired entry
> be removed from both memory and persistence?
>
> -Val
>
> On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > The ticket [1] is in patch available state looks good, the only thing
> left
> > there is to transfer old entries to new storage. I hope Andrey will have
> > time to finish this soon, so we can target the fix for 2.5.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> >
> > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> >
> > > Val,
> > >
> > > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> > >
> > > In general, the more deployments the persistence will get the more
> demand
> > > we will see for that capability. Personally, I'd create a ticket for
> now
> > > and put it off to our backlog.
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > > wrote:
> > >
> > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov  >
> > > > wrote:
> > > >
> > > > > As far as I know there is no plans. Denis please correct me if I'm
> > > wrong.
> > > > >
> > > > > But users found these or similar bugs, it seems we need to support
> > PDS
> > > > and
> > > > > TTL.
> > > > >
> > > >
> > > > We should either support it or throw a clear  exception on startup
> > > clearly
> > > > stating that it is not. I suppose, it should be the latter for now.
> > > > Dmitriy, any chance you can file a ticket for it?
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Valentin Kulichenko]

Alex,

What is behavior going to be after IGNITE-5874 is fixed? Will expired entry
be removed from both memory and persistence?

-Val

On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> The ticket [1] is in patch available state looks good, the only thing left
> there is to transfer old entries to new storage. I hope Andrey will have
> time to finish this soon, so we can target the fix for 2.5.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5874
>
> 2018-03-09 22:51 GMT+03:00 Denis Magda :
>
> > Val,
> >
> > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> >
> > In general, the more deployments the persistence will get the more demand
> > we will see for that capability. Personally, I'd create a ticket for now
> > and put it off to our backlog.
> >
> > --
> > Denis
> >
> >
> > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan  >
> > wrote:
> >
> > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov 
> > > wrote:
> > >
> > > > As far as I know there is no plans. Denis please correct me if I'm
> > wrong.
> > > >
> > > > But users found these or similar bugs, it seems we need to support
> PDS
> > > and
> > > > TTL.
> > > >
> > >
> > > We should either support it or throw a clear  exception on startup
> > clearly
> > > stating that it is not. I suppose, it should be the latter for now.
> > > Dmitriy, any chance you can file a ticket for it?
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Dmitry Pavlov]

Hi Denis,

yes, this is about PDS + entry expiration (TTL).

Please note there is second issue, than is visible by user
https://issues.apache.org/jira/browse/IGNITE-6964

'transfer of old entries to new storage' means DB entries migration from
old format to new. We defenetely should have PDS compatiblity test for this
case.

Sincerely,
Dmitriy Pavlov

вс, 11 мар. 2018 г. в 11:06, Denis Magda :

> Alex,
>
> Is that ticket about the eviction from the persistence layer? It's not
> obvious from the description.
>
> Also, what do you mean by the "transfer of old entries to new storage" (how
> is it intended to work and what's a rationale if you're talking about pages
> movement from Ignite persistence to another)?
>
> --
> Denis
>
> On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > The ticket [1] is in patch available state looks good, the only thing
> left
> > there is to transfer old entries to new storage. I hope Andrey will have
> > time to finish this soon, so we can target the fix for 2.5.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5874
> >
> > 2018-03-09 22:51 GMT+03:00 Denis Magda :
> >
> > > Val,
> > >
> > > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> > >
> > > In general, the more deployments the persistence will get the more
> demand
> > > we will see for that capability. Personally, I'd create a ticket for
> now
> > > and put it off to our backlog.
> > >
> > > --
> > > Denis
> > >
> > >
> > > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > > wrote:
> > >
> > > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov  >
> > > > wrote:
> > > >
> > > > > As far as I know there is no plans. Denis please correct me if I'm
> > > wrong.
> > > > >
> > > > > But users found these or similar bugs, it seems we need to support
> > PDS
> > > > and
> > > > > TTL.
> > > > >
> > > >
> > > > We should either support it or throw a clear  exception on startup
> > > clearly
> > > > stating that it is not. I suppose, it should be the latter for now.
> > > > Dmitriy, any chance you can file a ticket for it?
> > > >
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Denis Magda]

Alex,

Is that ticket about the eviction from the persistence layer? It's not
obvious from the description.

Also, what do you mean by the "transfer of old entries to new storage" (how
is it intended to work and what's a rationale if you're talking about pages
movement from Ignite persistence to another)?

--
Denis

On Sat, Mar 10, 2018 at 12:06 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> The ticket [1] is in patch available state looks good, the only thing left
> there is to transfer old entries to new storage. I hope Andrey will have
> time to finish this soon, so we can target the fix for 2.5.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5874
>
> 2018-03-09 22:51 GMT+03:00 Denis Magda :
>
> > Val,
> >
> > I'd like to hear Alexey G. opinion on this? Alex, please chime in.
> >
> > In general, the more deployments the persistence will get the more demand
> > we will see for that capability. Personally, I'd create a ticket for now
> > and put it off to our backlog.
> >
> > --
> > Denis
> >
> >
> > On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan  >
> > wrote:
> >
> > > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov 
> > > wrote:
> > >
> > > > As far as I know there is no plans. Denis please correct me if I'm
> > wrong.
> > > >
> > > > But users found these or similar bugs, it seems we need to support
> PDS
> > > and
> > > > TTL.
> > > >
> > >
> > > We should either support it or throw a clear  exception on startup
> > clearly
> > > stating that it is not. I suppose, it should be the latter for now.
> > > Dmitriy, any chance you can file a ticket for it?
> > >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Alexey Goncharuk]

The ticket [1] is in patch available state looks good, the only thing left
there is to transfer old entries to new storage. I hope Andrey will have
time to finish this soon, so we can target the fix for 2.5.

[1] https://issues.apache.org/jira/browse/IGNITE-5874

2018-03-09 22:51 GMT+03:00 Denis Magda :

> Val,
>
> I'd like to hear Alexey G. opinion on this? Alex, please chime in.
>
> In general, the more deployments the persistence will get the more demand
> we will see for that capability. Personally, I'd create a ticket for now
> and put it off to our backlog.
>
> --
> Denis
>
>
> On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan 
> wrote:
>
> > On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov 
> > wrote:
> >
> > > As far as I know there is no plans. Denis please correct me if I'm
> wrong.
> > >
> > > But users found these or similar bugs, it seems we need to support PDS
> > and
> > > TTL.
> > >
> >
> > We should either support it or throw a clear  exception on startup
> clearly
> > stating that it is not. I suppose, it should be the latter for now.
> > Dmitriy, any chance you can file a ticket for it?
> >
>

Re: Data eviction/expiration from Ignite persistence

[Denis Magda]

Val,

I'd like to hear Alexey G. opinion on this? Alex, please chime in.

In general, the more deployments the persistence will get the more demand
we will see for that capability. Personally, I'd create a ticket for now
and put it off to our backlog.

--
Denis


On Fri, Mar 9, 2018 at 9:34 AM, Dmitriy Setrakyan 
wrote:

> On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov 
> wrote:
>
> > As far as I know there is no plans. Denis please correct me if I'm wrong.
> >
> > But users found these or similar bugs, it seems we need to support PDS
> and
> > TTL.
> >
>
> We should either support it or throw a clear  exception on startup clearly
> stating that it is not. I suppose, it should be the latter for now.
> Dmitriy, any chance you can file a ticket for it?
>

Re: Data eviction/expiration from Ignite persistence

[Dmitriy Setrakyan]

On Fri, Mar 9, 2018 at 3:43 AM, Dmitry Pavlov  wrote:

> As far as I know there is no plans. Denis please correct me if I'm wrong.
>
> But users found these or similar bugs, it seems we need to support PDS and
> TTL.
>

We should either support it or throw a clear  exception on startup clearly
stating that it is not. I suppose, it should be the latter for now.
Dmitriy, any chance you can file a ticket for it?

Re: Data eviction/expiration from Ignite persistence

[Dmitry Pavlov]

As far as I know there is no plans. Denis please correct me if I'm wrong.

But users found these or similar bugs, it seems we need to support PDS and
TTL.

пт, 9 мар. 2018 г., 4:36 Valentin Kulichenko :

> Guys,
>
> What is the result of this discussion? Do we still not support eviction and
> expiration on persistence level? If so, any plans to change this?
>
> -Val
>
> On Tue, Nov 21, 2017 at 9:45 AM, Denis Magda  wrote:
>
> > We might break the compatibility for the next major release or even
> create
> > a tool that will migrate persistence files from an old to new formats.
> >
> > —
> > Denis
> >
> > > On Nov 21, 2017, at 8:34 AM, Dmitry Pavlov 
> > wrote:
> > >
> > > Hi Denis,
> > >
> > > Second fix we need to do is B+ tree separation in per-partition basis:
> > > https://issues.apache.org/jira/browse/IGNITE-5874
> > >
> > > Should we take into account compatibilty issues with previous Ignite
> > > persistent store versions, because current TTL tree is persisted, and
> > will
> > > change its format?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > >
> > > вт, 21 нояб. 2017 г. в 2:13, Denis Magda :
> > >
> > >> Dmitriy,
> > >>
> > >> That’s about TTL and eviction support for Ignite persistence.
> Presently
> > if
> > >> you set an expiration or eviction policy for a cache it will be
> applied
> > for
> > >> data stored in memory. The policy never affects the persistence layer.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov 
> > >> wrote:
> > >>>
> > >>> Hi Denis,
> > >>>
> > >>> Is this need covered by PDS + TTL?
> > >>>
> > >>> For the very first TTL test, I found some delay after applying TTL
> with
> > >> the
> > >>> repository enabled:
> https://issues.apache.org/jira/browse/IGNITE-6964
> > >>>
> > >>> And I'm wondering if the user's needs are covered by
> > >>> https://apacheignite.readme.io/docs/expiry-policies plus
> > >>> https://apacheignite.readme.io/docs/distributed-persistent-store
> > >>>
> > >>> Sincerely,
> > >>> Dmitriy Pavlov
> > >>>
> > >>> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov :
> > >>>
> >  Hi Denis,
> > 
> >  What is the difference of required by users functionality with TTL
> > cache
> >  expiration?
> > 
> >  By some posts I can suppose TTL cache is compatible with native
> >  persistence.
> > 
> >  Sincerely,
> >  Dmitriy Pavlov
> > 
> >  сб, 18 нояб. 2017 г. в 0:41, Denis Magda :
> > 
> > > Igniters,
> > >
> > > I’ve been talking to many Ignite users here and there who are
> already
> > >> on
> > > Ignite persistence or consider to turn it on. The majority of them
> > are
> > >> more
> > > than satisfied with its current state and provided capabilities.
> > >> That’s is
> > > really good news for us.
> > >
> > > However, I tend to come across the people who ask about
> > > eviction/expiration policies for the persistence itself. Had
> around 6
> > > conversation about the topic this month only.
> > >
> > > Usually the requirement is connected with a streaming use case.
> When
> > an
> > > application streams a lot of data (IoT, metrics, etc.) to the
> cluster
> > >> but
> > > the data becomes stale in some period of time (day, couple of days,
> > >> etc.).
> > > The user doesn’t want to waste the disk space and needs to simple
> > >> purge the
> > > data from there.
> > >
> > > My suggestion here is to create a timer task that will remove the
> > stale
> > > data from the cluster. However, since the demand is growing
> probably
> > >> it’s a
> > > good time to discuss a feasibility of this feature.
> > >
> > > Alex G, as the main architect of the persistence, could you share
> > your
> > > thoughts on this? What will it cost to us to support
> > >> eviction/expiration
> > > for the persistence?
> > >
> > > —
> > > Denis
> > 
> > 
> > >>
> > >>
> >
> >
>

Re: Data eviction/expiration from Ignite persistence

[Valentin Kulichenko]

Guys,

What is the result of this discussion? Do we still not support eviction and
expiration on persistence level? If so, any plans to change this?

-Val

On Tue, Nov 21, 2017 at 9:45 AM, Denis Magda  wrote:

> We might break the compatibility for the next major release or even create
> a tool that will migrate persistence files from an old to new formats.
>
> —
> Denis
>
> > On Nov 21, 2017, at 8:34 AM, Dmitry Pavlov 
> wrote:
> >
> > Hi Denis,
> >
> > Second fix we need to do is B+ tree separation in per-partition basis:
> > https://issues.apache.org/jira/browse/IGNITE-5874
> >
> > Should we take into account compatibilty issues with previous Ignite
> > persistent store versions, because current TTL tree is persisted, and
> will
> > change its format?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> >
> > вт, 21 нояб. 2017 г. в 2:13, Denis Magda :
> >
> >> Dmitriy,
> >>
> >> That’s about TTL and eviction support for Ignite persistence. Presently
> if
> >> you set an expiration or eviction policy for a cache it will be applied
> for
> >> data stored in memory. The policy never affects the persistence layer.
> >>
> >> —
> >> Denis
> >>
> >>> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov 
> >> wrote:
> >>>
> >>> Hi Denis,
> >>>
> >>> Is this need covered by PDS + TTL?
> >>>
> >>> For the very first TTL test, I found some delay after applying TTL with
> >> the
> >>> repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
> >>>
> >>> And I'm wondering if the user's needs are covered by
> >>> https://apacheignite.readme.io/docs/expiry-policies plus
> >>> https://apacheignite.readme.io/docs/distributed-persistent-store
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov :
> >>>
>  Hi Denis,
> 
>  What is the difference of required by users functionality with TTL
> cache
>  expiration?
> 
>  By some posts I can suppose TTL cache is compatible with native
>  persistence.
> 
>  Sincerely,
>  Dmitriy Pavlov
> 
>  сб, 18 нояб. 2017 г. в 0:41, Denis Magda :
> 
> > Igniters,
> >
> > I’ve been talking to many Ignite users here and there who are already
> >> on
> > Ignite persistence or consider to turn it on. The majority of them
> are
> >> more
> > than satisfied with its current state and provided capabilities.
> >> That’s is
> > really good news for us.
> >
> > However, I tend to come across the people who ask about
> > eviction/expiration policies for the persistence itself. Had around 6
> > conversation about the topic this month only.
> >
> > Usually the requirement is connected with a streaming use case. When
> an
> > application streams a lot of data (IoT, metrics, etc.) to the cluster
> >> but
> > the data becomes stale in some period of time (day, couple of days,
> >> etc.).
> > The user doesn’t want to waste the disk space and needs to simple
> >> purge the
> > data from there.
> >
> > My suggestion here is to create a timer task that will remove the
> stale
> > data from the cluster. However, since the demand is growing probably
> >> it’s a
> > good time to discuss a feasibility of this feature.
> >
> > Alex G, as the main architect of the persistence, could you share
> your
> > thoughts on this? What will it cost to us to support
> >> eviction/expiration
> > for the persistence?
> >
> > —
> > Denis
> 
> 
> >>
> >>
>
>

Re: Data eviction/expiration from Ignite persistence

[Denis Magda]

We might break the compatibility for the next major release or even create a 
tool that will migrate persistence files from an old to new formats.

—
Denis

> On Nov 21, 2017, at 8:34 AM, Dmitry Pavlov  wrote:
> 
> Hi Denis,
> 
> Second fix we need to do is B+ tree separation in per-partition basis:
> https://issues.apache.org/jira/browse/IGNITE-5874
> 
> Should we take into account compatibilty issues with previous Ignite
> persistent store versions, because current TTL tree is persisted, and will
> change its format?
> 
> Sincerely,
> Dmitriy Pavlov
> 
> 
> вт, 21 нояб. 2017 г. в 2:13, Denis Magda :
> 
>> Dmitriy,
>> 
>> That’s about TTL and eviction support for Ignite persistence. Presently if
>> you set an expiration or eviction policy for a cache it will be applied for
>> data stored in memory. The policy never affects the persistence layer.
>> 
>> —
>> Denis
>> 
>>> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov 
>> wrote:
>>> 
>>> Hi Denis,
>>> 
>>> Is this need covered by PDS + TTL?
>>> 
>>> For the very first TTL test, I found some delay after applying TTL with
>> the
>>> repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
>>> 
>>> And I'm wondering if the user's needs are covered by
>>> https://apacheignite.readme.io/docs/expiry-policies plus
>>> https://apacheignite.readme.io/docs/distributed-persistent-store
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov :
>>> 
 Hi Denis,
 
 What is the difference of required by users functionality with TTL cache
 expiration?
 
 By some posts I can suppose TTL cache is compatible with native
 persistence.
 
 Sincerely,
 Dmitriy Pavlov
 
 сб, 18 нояб. 2017 г. в 0:41, Denis Magda :
 
> Igniters,
> 
> I’ve been talking to many Ignite users here and there who are already
>> on
> Ignite persistence or consider to turn it on. The majority of them are
>> more
> than satisfied with its current state and provided capabilities.
>> That’s is
> really good news for us.
> 
> However, I tend to come across the people who ask about
> eviction/expiration policies for the persistence itself. Had around 6
> conversation about the topic this month only.
> 
> Usually the requirement is connected with a streaming use case. When an
> application streams a lot of data (IoT, metrics, etc.) to the cluster
>> but
> the data becomes stale in some period of time (day, couple of days,
>> etc.).
> The user doesn’t want to waste the disk space and needs to simple
>> purge the
> data from there.
> 
> My suggestion here is to create a timer task that will remove the stale
> data from the cluster. However, since the demand is growing probably
>> it’s a
> good time to discuss a feasibility of this feature.
> 
> Alex G, as the main architect of the persistence, could you share your
> thoughts on this? What will it cost to us to support
>> eviction/expiration
> for the persistence?
> 
> —
> Denis
 
 
>> 
>> 

Re: Data eviction/expiration from Ignite persistence

[Dmitry Pavlov]

Hi Denis,

Second fix we need to do is B+ tree separation in per-partition basis:
https://issues.apache.org/jira/browse/IGNITE-5874

Should we take into account compatibilty issues with previous Ignite
persistent store versions, because current TTL tree is persisted, and will
change its format?

Sincerely,
Dmitriy Pavlov


вт, 21 нояб. 2017 г. в 2:13, Denis Magda :

> Dmitriy,
>
> That’s about TTL and eviction support for Ignite persistence. Presently if
> you set an expiration or eviction policy for a cache it will be applied for
> data stored in memory. The policy never affects the persistence layer.
>
> —
> Denis
>
> > On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov 
> wrote:
> >
> > Hi Denis,
> >
> > Is this need covered by PDS + TTL?
> >
> > For the very first TTL test, I found some delay after applying TTL with
> the
> > repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
> >
> > And I'm wondering if the user's needs are covered by
> > https://apacheignite.readme.io/docs/expiry-policies plus
> > https://apacheignite.readme.io/docs/distributed-persistent-store
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov :
> >
> >> Hi Denis,
> >>
> >> What is the difference of required by users functionality with TTL cache
> >> expiration?
> >>
> >> By some posts I can suppose TTL cache is compatible with native
> >> persistence.
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> сб, 18 нояб. 2017 г. в 0:41, Denis Magda :
> >>
> >>> Igniters,
> >>>
> >>> I’ve been talking to many Ignite users here and there who are already
> on
> >>> Ignite persistence or consider to turn it on. The majority of them are
> more
> >>> than satisfied with its current state and provided capabilities.
> That’s is
> >>> really good news for us.
> >>>
> >>> However, I tend to come across the people who ask about
> >>> eviction/expiration policies for the persistence itself. Had around 6
> >>> conversation about the topic this month only.
> >>>
> >>> Usually the requirement is connected with a streaming use case. When an
> >>> application streams a lot of data (IoT, metrics, etc.) to the cluster
> but
> >>> the data becomes stale in some period of time (day, couple of days,
> etc.).
> >>> The user doesn’t want to waste the disk space and needs to simple
> purge the
> >>> data from there.
> >>>
> >>> My suggestion here is to create a timer task that will remove the stale
> >>> data from the cluster. However, since the demand is growing probably
> it’s a
> >>> good time to discuss a feasibility of this feature.
> >>>
> >>> Alex G, as the main architect of the persistence, could you share your
> >>> thoughts on this? What will it cost to us to support
> eviction/expiration
> >>> for the persistence?
> >>>
> >>> —
> >>> Denis
> >>
> >>
>
>

Re: Data eviction/expiration from Ignite persistence

[Denis Magda]

Dmitriy,

That’s about TTL and eviction support for Ignite persistence. Presently if you 
set an expiration or eviction policy for a cache it will be applied for data 
stored in memory. The policy never affects the persistence layer.

—
Denis

> On Nov 20, 2017, at 9:29 AM, Dmitry Pavlov  wrote:
> 
> Hi Denis,
> 
> Is this need covered by PDS + TTL?
> 
> For the very first TTL test, I found some delay after applying TTL with the
> repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964
> 
> And I'm wondering if the user's needs are covered by
> https://apacheignite.readme.io/docs/expiry-policies plus
> https://apacheignite.readme.io/docs/distributed-persistent-store
> 
> Sincerely,
> Dmitriy Pavlov
> 
> сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov :
> 
>> Hi Denis,
>> 
>> What is the difference of required by users functionality with TTL cache
>> expiration?
>> 
>> By some posts I can suppose TTL cache is compatible with native
>> persistence.
>> 
>> Sincerely,
>> Dmitriy Pavlov
>> 
>> сб, 18 нояб. 2017 г. в 0:41, Denis Magda :
>> 
>>> Igniters,
>>> 
>>> I’ve been talking to many Ignite users here and there who are already on
>>> Ignite persistence or consider to turn it on. The majority of them are more
>>> than satisfied with its current state and provided capabilities. That’s is
>>> really good news for us.
>>> 
>>> However, I tend to come across the people who ask about
>>> eviction/expiration policies for the persistence itself. Had around 6
>>> conversation about the topic this month only.
>>> 
>>> Usually the requirement is connected with a streaming use case. When an
>>> application streams a lot of data (IoT, metrics, etc.) to the cluster but
>>> the data becomes stale in some period of time (day, couple of days, etc.).
>>> The user doesn’t want to waste the disk space and needs to simple purge the
>>> data from there.
>>> 
>>> My suggestion here is to create a timer task that will remove the stale
>>> data from the cluster. However, since the demand is growing probably it’s a
>>> good time to discuss a feasibility of this feature.
>>> 
>>> Alex G, as the main architect of the persistence, could you share your
>>> thoughts on this? What will it cost to us to support eviction/expiration
>>> for the persistence?
>>> 
>>> —
>>> Denis
>> 
>> 

Re: Data eviction/expiration from Ignite persistence

[Dmitry Pavlov]

Hi Denis,

Is this need covered by PDS + TTL?

For the very first TTL test, I found some delay after applying TTL with the
repository enabled: https://issues.apache.org/jira/browse/IGNITE-6964

And I'm wondering if the user's needs are covered by
https://apacheignite.readme.io/docs/expiry-policies plus
https://apacheignite.readme.io/docs/distributed-persistent-store

Sincerely,
Dmitriy Pavlov

сб, 18 нояб. 2017 г. в 12:12, Dmitry Pavlov :

> Hi Denis,
>
> What is the difference of required by users functionality with TTL cache
> expiration?
>
> By some posts I can suppose TTL cache is compatible with native
> persistence.
>
> Sincerely,
> Dmitriy Pavlov
>
> сб, 18 нояб. 2017 г. в 0:41, Denis Magda :
>
>> Igniters,
>>
>> I’ve been talking to many Ignite users here and there who are already on
>> Ignite persistence or consider to turn it on. The majority of them are more
>> than satisfied with its current state and provided capabilities. That’s is
>> really good news for us.
>>
>> However, I tend to come across the people who ask about
>> eviction/expiration policies for the persistence itself. Had around 6
>> conversation about the topic this month only.
>>
>> Usually the requirement is connected with a streaming use case. When an
>> application streams a lot of data (IoT, metrics, etc.) to the cluster but
>> the data becomes stale in some period of time (day, couple of days, etc.).
>> The user doesn’t want to waste the disk space and needs to simple purge the
>> data from there.
>>
>> My suggestion here is to create a timer task that will remove the stale
>> data from the cluster. However, since the demand is growing probably it’s a
>> good time to discuss a feasibility of this feature.
>>
>> Alex G, as the main architect of the persistence, could you share your
>> thoughts on this? What will it cost to us to support eviction/expiration
>> for the persistence?
>>
>> —
>> Denis
>
>

Re: Data eviction/expiration from Ignite persistence

[Dmitry Pavlov]

Hi Denis,

What is the difference of required by users functionality with TTL cache
expiration?

By some posts I can suppose TTL cache is compatible with native
persistence.

Sincerely,
Dmitriy Pavlov

сб, 18 нояб. 2017 г. в 0:41, Denis Magda :

> Igniters,
>
> I’ve been talking to many Ignite users here and there who are already on
> Ignite persistence or consider to turn it on. The majority of them are more
> than satisfied with its current state and provided capabilities. That’s is
> really good news for us.
>
> However, I tend to come across the people who ask about
> eviction/expiration policies for the persistence itself. Had around 6
> conversation about the topic this month only.
>
> Usually the requirement is connected with a streaming use case. When an
> application streams a lot of data (IoT, metrics, etc.) to the cluster but
> the data becomes stale in some period of time (day, couple of days, etc.).
> The user doesn’t want to waste the disk space and needs to simple purge the
> data from there.
>
> My suggestion here is to create a timer task that will remove the stale
> data from the cluster. However, since the demand is growing probably it’s a
> good time to discuss a feasibility of this feature.
>
> Alex G, as the main architect of the persistence, could you share your
> thoughts on this? What will it cost to us to support eviction/expiration
> for the persistence?
>
> —
> Denis