Hi Vladimir,
It took me a while to make a small test case but I now have one (see below)
With the setExpiryPolicyFactory method enabled the heap after operation
983Mb, with the setExpiryPolicyFactory commented out it is just 85Mb.
Simple run this class file and once you see the console message attach
JVisualVM, then spam the "Perform GC" button.
Also without the expiry time set the test runs with -Xm512m and with
expiry the test needs -Xmx2024m.
Thanks
Neil
-- IgniteExpiryIssue.java --
package de.wightman.neil.apache.ignite.test;
import javax.cache.configuration.FactoryBuilder;
import javax.cache.expiry.CreatedExpiryPolicy;
import javax.cache.expiry.Duration;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteDataStreamer;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheMemoryMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.eviction.lru.LruEvictionPolicy;
import org.apache.ignite.cache.query.annotations.QuerySqlField;
import org.apache.ignite.configuration.CacheConfiguration;
public class IgniteExpiryIssue
{
public static void main( final String[] args ) throws
InterruptedException
{
final long start = System.currentTimeMillis();
// configures Tcp discovery for local host only
try (final Ignite ignite = Ignition.start( "LocalConfig.xml" ))
{
final IgniteCache<CustomKey, CustomValue> cache =
ignite.getOrCreateCache( cacheConfig() );
final IgniteDataStreamer<Object, Object> metricStreamer =
ignite.dataStreamer( "TestCacheName" );
final long counter = 1000;
final long counter2 = 49;
final long minutes = 60;
final long fakeBaseTime = System.currentTimeMillis();
for( long k = 0; k < minutes; k++ )
{
final long ts = fakeBaseTime + (minutes * 60 * 1000);
for( long i = 0; i < counter; i++ )
{
for( long j = 0; j < counter2; j++ )
{
final CustomKey key = new CustomKey( ts, i, j );
final CustomValue value = new CustomValue( 1.0,
1.0, 1.0, 1.0, 10 );
metricStreamer.addData( key, value );
}
}
}
// debug memory issue here
System.out.println("Duration = " +
(System.currentTimeMillis() - start));
System.out.println("Cache filled attach JVisualVM to view
memory and generate a heap dump.");
Thread.sleep(Long.MAX_VALUE);
}
}
public static CacheConfiguration<CustomKey, CustomValue> cacheConfig()
{
final CacheConfiguration<CustomKey, CustomValue>
cacheConfiguration = new CacheConfiguration<>( "TestCacheName" );
cacheConfiguration.setBackups( 1 );
cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
cacheConfiguration.setIndexedTypes( CustomKey.class,
CustomValue.class );
cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L *
1024L );
cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy(
6_000_000 ) );
cacheConfiguration.setSwapEnabled( true );
cacheConfiguration.setStatisticsEnabled( true );
// Causes heap usage to be high
cacheConfiguration.setExpiryPolicyFactory(
FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
return cacheConfiguration;
}
public static class CustomKey
{
@QuerySqlField(index = true, descending = false, name = "ts")
private long timeStamp;
@QuerySqlField(index = true, name = "id1")
private long id1;
@QuerySqlField(index = true, name = "id2")
private long id2;
public CustomKey( long timeStamp,
long id1,
long id2 )
{
this.timeStamp = timeStamp;
this.id1 = id1;
this.id2 = id2;
}
}
public static class CustomValue
{
@QuerySqlField(name = "_d1")
private double d1;
@QuerySqlField()
private double d2;
@QuerySqlField()
private double d3;
@QuerySqlField()
private double d4;
@QuerySqlField(name = "_count")
private long count;
public CustomValue( double d1,
double d2,
double d3,
double d4,
long count )
{
this.d1 = d1;
this.d2 = d2;
this.d3 = d3;
this.d4 = d4;
this.count = count;
}
}
}
----
On 10/03/16 13:04, Vladimir Ozerov wrote:
Hi Neil,
Could you please attach the code reproducing the problem?
Vladimir.
On Wed, Mar 9, 2016 at 1:16 PM, Neil Wightman
<[email protected] <mailto:[email protected]>> wrote:
Hi All,
I have been trying out ignite for the past few weeks but I am
hitting a strange problem. I dont know if this problem is in my
code or ignite.
Currently I have an OFFHEAP_TIERED cache with ~3 million entries,
max off heap size of 4Gb, but this is showing some very high heap
memory usage. The cache stores data for 1 hour then expires it
and constantly streams data into this cache via the
dataStreamer.addData API.
The problem seems to be with this specific expiry policy
configuration :
cacheConfiguration.setExpiryPolicyFactory(
FactoryBuilder.factoryOf( new CreatedExpiryPolicy(
Duration.ONE_HOUR ) ) );
- with this expiry policy the heap memory usage for the ignite
process is 976Mb, without this expiry policy set its just 156Mb.
The actual off heap memory usage is only 557Mb (assuming the cache
metrics are correct).
I have generated two heap dumps and the memory difference is all
in the GridCacheTtlManager which is retaining a heap of ~889Mb
according to Eclipses MAT.
I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT
from git (Monday 7th March) and both show the same issue.
The problem with this large heap usage is that I am now getting
very large GC pauses which I cant stop without disabling the
expiry policy.
My main question is am I doing something wrong? Is the Expiry
Policy really supposed to use so much heap memory compared to the
off heap cache size?
Thanks in advance,
Neil
---- config definition ----
final CacheConfiguration<MetricKey, AggregatePack>
cacheConfiguration = new CacheConfiguration<>( "cacheName" );
cacheConfiguration.setBackups( 1 );
cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
cacheConfiguration.setIndexedTypes( CustomKey.class,
CustomData.class );
cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>(
6_000_000 ) );
cacheConfiguration.setSwapEnabled( true );
cacheConfiguration.setStatisticsEnabled( true );
// This causes large memory usage
cacheConfiguration.setExpiryPolicyFactory(
FactoryBuilder.factoryOf( new CreatedExpiryPolicy(
Duration.ONE_HOUR ) ) );
