Re: Memory policies examples fails

[Denis Magda]

Sergey,

I tweaked the example following your suggestion. Now it works, thanks.

In addition, a javadoc of MemoryConfiguraiton, MemoryPolicyConfiguration and 
DataPageEvictionMode has been significantly improved. There are some of the 
parameters like page memory’s concurrency mode and policy’s "empty page pool 
size" that are not obvious. Marked all the blur places in those configurations 
with TODOs and created a ticket [1]. * Alex G *, as an architect of the page 
memory, please look at [1] and try to close it tomorrow so that I can complete 
the readme part.

Finally, as for the out of memory exception I fully support Dmitriy’s opinion - 
let’s be as more concrete as possible. "Failed to allocate new page within 
 MemoryPolicy” sounds to generic and doesn’t help me anyhow. 
Let’s add to the message what was a root cause (policy size, name, etc.) so 
that the user can fix it immediately. Created a ticket [2].

[1] https://issues.apache.org/jira/browse/IGNITE-5007 

[2] https://issues.apache.org/jira/browse/IGNITE-5008

—
Denis

> On Apr 17, 2017, at 2:18 PM, Dmitriy Setrakyan  wrote:
> 
> On Mon, Apr 17, 2017 at 9:06 AM, Sergey Chugunov 
> wrote:
> 
>> Dmitriy,
>> 
>> I think this issue highlights another lack in our documentation that
>> emerged with introduction of PageMemory.
>> 
>> In current documentation we have some recommendations about memory capacity
>> planning.
>> As PageMemory storage data structures introduce their own overhead; this
>> section of documentation must be updated.
>> 
>> As per "null" in OutOfMemoryException message, I've already addressed it so
>> now when PageMemory fails to allocate new page, it throws an
>> IgniteOutOfMemoryException with message like "Failed to allocate new page
>> within  MemoryPolicy".
>> 
> 
> Sergey, the error message should also mention which configuration property
> needs to be changed to fix the issue.

Re: Memory policies examples fails

[Dmitriy Setrakyan]

On Mon, Apr 17, 2017 at 9:06 AM, Sergey Chugunov 
wrote:

> Dmitriy,
>
> I think this issue highlights another lack in our documentation that
> emerged with introduction of PageMemory.
>
> In current documentation we have some recommendations about memory capacity
> planning.
> As PageMemory storage data structures introduce their own overhead; this
> section of documentation must be updated.
>
> As per "null" in OutOfMemoryException message, I've already addressed it so
> now when PageMemory fails to allocate new page, it throws an
> IgniteOutOfMemoryException with message like "Failed to allocate new page
> within  MemoryPolicy".
>

Sergey, the error message should also mention which configuration property
needs to be changed to fix the issue.

Re: Memory policies examples fails

[Sergey Chugunov]

Dmitriy,

I think this issue highlights another lack in our documentation that
emerged with introduction of PageMemory.

In current documentation we have some recommendations about memory capacity
planning.
As PageMemory storage data structures introduce their own overhead; this
section of documentation must be updated.

As per "null" in OutOfMemoryException message, I've already addressed it so
now when PageMemory fails to allocate new page, it throws an
IgniteOutOfMemoryException with message like "Failed to allocate new page
within  MemoryPolicy".


Thanks,
Sergey.

On Mon, Apr 17, 2017 at 6:52 PM, Dmitriy Setrakyan 
wrote:

> Wow If it happened to Denis, it will happen to many other users as
> well. With the exception error message saying "Null", I doubt any user
> would figure out what to do.
>
> Can we fix the error message?
>
> D.
>
>
> On Mon, Apr 17, 2017 at 6:03 AM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > Denis,
> >
> > I found an issue with the example: each cache introduces some memory
> > overhead (mostly for cache partitions data structures) and it must be
> > counted in MemoryPolicy size.
> >
> > To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
> > and *5MB_Region_Swapping *to 10 MB.
> >
> > I didn't find any documentation on this, but I think it is part of new
> > PageMemory-based storage so it needs to be documented as part of
> ignite-2.0
> > release.
> >
> > Thanks,
> > Sergey.
> >
> >
> >
> > On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov <
> > sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Denis,
> > >
> > > It looks like partition exchange mechanism allocates a lot of pages
> right
> > > after the cache is started.
> > > I'm trying to figure out what's going on.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > > On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda 
> wrote:
> > >
> > >> Igniters,
> > >>
> > >> I’ve created a simple example to demonstrate how to set up various
> > >> policies:
> > >> https://github.com/apache/ignite/blob/master/examples/src/
> > >> main/java/org/apache/ignite/examples/datagrid/MemoryPolicies
> > Example.java
> > >>
> > >> However, this tiny example fails with
> > >>
> > >> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
> > >> null
> > >> at org.apache.ignite.internal.pagemem.impl.
> PageMemoryNoStoreImp
> > >> l.allocatePage(PageMemoryNoStoreImpl.java:255)
> > >> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
> > >> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
> > >> rImpl.java:727)
> > >> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
> > >> Impl.java:711)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
> > >> nTopologyImpl.java:718)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
> > >> nTopologyImpl.java:405)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
> > >> TopologyImpl.java:569)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> preloader.GridDhtPartitionsExchangeFuture.
> > distributedExchange(GridDhtPar
> > >> titionsExchangeFuture.java:844)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
> > >> ionsExchangeFuture.java:573)
> > >> at org.apache.ignite.internal.processors.cache.
> GridCachePartiti
> > >> onExchangeManager$ExchangeWorker.body(GridCacheP
> > >> artitionExchangeManager.java:1800)
> > >> ... 2 more
> > >>
> > >>
> > >>
> > >> What I’m doing wrong? Alex G., could you take a look? Probably I’m
> > >> missing something due to a lack of documentation.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>
> > >
> >
>

Re: Memory policies examples fails

[Dmitriy Setrakyan]

Wow If it happened to Denis, it will happen to many other users as
well. With the exception error message saying "Null", I doubt any user
would figure out what to do.

Can we fix the error message?

D.


On Mon, Apr 17, 2017 at 6:03 AM, Sergey Chugunov 
wrote:

> Denis,
>
> I found an issue with the example: each cache introduces some memory
> overhead (mostly for cache partitions data structures) and it must be
> counted in MemoryPolicy size.
>
> To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
> and *5MB_Region_Swapping *to 10 MB.
>
> I didn't find any documentation on this, but I think it is part of new
> PageMemory-based storage so it needs to be documented as part of ignite-2.0
> release.
>
> Thanks,
> Sergey.
>
>
>
> On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > Denis,
> >
> > It looks like partition exchange mechanism allocates a lot of pages right
> > after the cache is started.
> > I'm trying to figure out what's going on.
> >
> > Thanks,
> > Sergey.
> >
> > On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda  wrote:
> >
> >> Igniters,
> >>
> >> I’ve created a simple example to demonstrate how to set up various
> >> policies:
> >> https://github.com/apache/ignite/blob/master/examples/src/
> >> main/java/org/apache/ignite/examples/datagrid/MemoryPolicies
> Example.java
> >>
> >> However, this tiny example fails with
> >>
> >> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
> >> null
> >> at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImp
> >> l.allocatePage(PageMemoryNoStoreImpl.java:255)
> >> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
> >> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
> >> rImpl.java:727)
> >> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
> >> Impl.java:711)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
> >> nTopologyImpl.java:718)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
> >> nTopologyImpl.java:405)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
> >> TopologyImpl.java:569)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> preloader.GridDhtPartitionsExchangeFuture.
> distributedExchange(GridDhtPar
> >> titionsExchangeFuture.java:844)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
> >> ionsExchangeFuture.java:573)
> >> at org.apache.ignite.internal.processors.cache.GridCachePartiti
> >> onExchangeManager$ExchangeWorker.body(GridCacheP
> >> artitionExchangeManager.java:1800)
> >> ... 2 more
> >>
> >>
> >>
> >> What I’m doing wrong? Alex G., could you take a look? Probably I’m
> >> missing something due to a lack of documentation.
> >>
> >> —
> >> Denis
> >>
> >>
> >
>

Re: Memory policies examples fails

[Sergey Chugunov]

Denis,

I found an issue with the example: each cache introduces some memory
overhead (mostly for cache partitions data structures) and it must be
counted in MemoryPolicy size.

To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
and *5MB_Region_Swapping *to 10 MB.

I didn't find any documentation on this, but I think it is part of new
PageMemory-based storage so it needs to be documented as part of ignite-2.0
release.

Thanks,
Sergey.



On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov 
wrote:

> Denis,
>
> It looks like partition exchange mechanism allocates a lot of pages right
> after the cache is started.
> I'm trying to figure out what's going on.
>
> Thanks,
> Sergey.
>
> On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda  wrote:
>
>> Igniters,
>>
>> I’ve created a simple example to demonstrate how to set up various
>> policies:
>> https://github.com/apache/ignite/blob/master/examples/src/
>> main/java/org/apache/ignite/examples/datagrid/MemoryPoliciesExample.java
>>
>> However, this tiny example fails with
>>
>> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
>> null
>> at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImp
>> l.allocatePage(PageMemoryNoStoreImpl.java:255)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
>> rImpl.java:727)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
>> Impl.java:711)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
>> nTopologyImpl.java:718)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
>> nTopologyImpl.java:405)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
>> TopologyImpl.java:569)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPar
>> titionsExchangeFuture.java:844)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
>> ionsExchangeFuture.java:573)
>> at org.apache.ignite.internal.processors.cache.GridCachePartiti
>> onExchangeManager$ExchangeWorker.body(GridCacheP
>> artitionExchangeManager.java:1800)
>> ... 2 more
>>
>>
>>
>> What I’m doing wrong? Alex G., could you take a look? Probably I’m
>> missing something due to a lack of documentation.
>>
>> —
>> Denis
>>
>>
>

Re: Memory policies examples fails

[Sergey Chugunov]

Denis,

It looks like partition exchange mechanism allocates a lot of pages right
after the cache is started.
I'm trying to figure out what's going on.

Thanks,
Sergey.

On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda  wrote:

> Igniters,
>
> I’ve created a simple example to demonstrate how to set up various
> policies:
> https://github.com/apache/ignite/blob/master/examples/
> src/main/java/org/apache/ignite/examples/datagrid/
> MemoryPoliciesExample.java
>
> However, this tiny example fails with
>
> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException: null
> at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.
> allocatePage(PageMemoryNoStoreImpl.java:255)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.allocateForTree(
> IgniteCacheOffheapManagerImpl.java:635)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.createCacheDataStore0(
> IgniteCacheOffheapManagerImpl.java:727)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.createCacheDataStore(
> IgniteCacheOffheapManagerImpl.java:711)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
> at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitionTopologyImpl.
> java:718)
> at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitionTopologyImpl.
> java:405)
> at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.
> java:569)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(
> GridDhtPartitionsExchangeFuture.java:844)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.preloader.GridDhtPartitionsExchangeFuture.init(
> GridDhtPartitionsExchangeFuture.java:573)
> at org.apache.ignite.internal.processors.cache.
> GridCachePartitionExchangeManager$ExchangeWorker.body(
> GridCachePartitionExchangeManager.java:1800)
> ... 2 more
>
>
>
> What I’m doing wrong? Alex G., could you take a look? Probably I’m missing
> something due to a lack of documentation.
>
> —
> Denis
>
>

Memory policies examples fails

[Denis Magda]

Igniters,

I’ve created a simple example to demonstrate how to set up various policies:
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/MemoryPoliciesExample.java

However, this tiny example fails with 

Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException: null
at 
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:255)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.createCacheDataStore0(IgniteCacheOffheapManagerImpl.java:727)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.createCacheDataStore(IgniteCacheOffheapManagerImpl.java:711)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitionTopologyImpl.java:718)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitionTopologyImpl.java:405)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.java:569)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:844)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:573)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1800)
... 2 more



What I’m doing wrong? Alex G., could you take a look? Probably I’m missing 
something due to a lack of documentation.

—
Denis