Hi Denis,
This is the code, more things:
#1 We store a *table* in each cache entry.
#2 Table has several columns, column may be a big object like more than 10k
bytes. Column implement interface Binarylizable.
I think you can try to produce this issue by design a class like above table.
Also if you need heap dump, I can share it also.
Thanks
Shawn
private void save(Context context)
{
BinaryObjectBuilder builder =
IgniteManager.getIgnite().binary().builder(context.cacheName);
builder.setField(IgniteConstants.COLUMN_TIMESTAMP,
context.table.getTimestamp() / 1000);
builder.setField(IgniteConstants.COLUMN_SITE, context.table.getSite());
builder.setField(IgniteConstants.COLUMN_PRODUCT,
context.table.getProduct());
context.table.getDimColumns().forEach(c -> builder.setField(c.getName(),
c));
builder.setField(context.table.getMeasColumn().getName(),
context.table.getMeasColumn());
IgniteCache<String, BinaryObject> cache = getCache(context);
cache.put(generateKey(context.table), builder.build());
}
private CacheConfiguration<String, BinaryObject>
createCacheConfiguration(Context context)
{
CacheConfiguration<String, BinaryObject> config = new
CacheConfiguration<>();
List<ColumnScheme> columns = new ArrayList<>();
columns.add(new ColumnScheme(IgniteConstants.COLUMN_TIMESTAMP,
Long.class.getTypeName(),
commonConfig.indexingTimestampEnable, false));
columns.add(new ColumnScheme(IgniteConstants.COLUMN_SITE,
commonConfig.indexingSiteEnable));
columns.add(new ColumnScheme(IgniteConstants.COLUMN_PRODUCT,
commonConfig.indexingProductEnable));
for (Column<String> column : context.table.getDimColumns())
{
columns.add(new ColumnScheme(column.getName(),
Column.class.getTypeName(), false));
}
String measName = context.table.getMeasColumn().getName();
columns.add(new ColumnScheme(measName, Column.class.getTypeName(), false));
config.setQueryEntities(Collections
.singleton(IgniteManager.createEntity(String.class.getTypeName(),
context.cacheName, columns)));
config.setName(context.cacheName);
config.setMemoryMode(commonConfig.cacheMemoryMode);
config.setBackups(commonConfig.backups);
config.setStartSize(10_000);
config.setCopyOnRead(commonConfig.copyOnRead);
return config;
}
private IgniteCache<String, BinaryObject> getCache(Context context)
{
IgniteCache<String, BinaryObject> cache =
IgniteManager.getIgnite().cache(context.cacheName);
if (cache == null)
{
cache =
IgniteManager.getIgnite().getOrCreateCache(createCacheConfiguration(context));
}
if (context.ttl > 0)
{
cache = cache.withExpiryPolicy(new ModifiedExpiryPolicy(new
Duration(TimeUnit.SECONDS, context.ttl)));
}
cache = cache.withKeepBinary();
return cache;
}
static class Context
{
final long ttl;
final Table table;
final String cacheName;
Context(long ttl, Table table)
{
this.ttl = ttl;
this.table = table;
this.cacheName = table.getSchema().generateCacheName();
}
}
发件人: Denis Magda [mailto:[email protected]]
发送时间: 2017年1月24日 3:30
收件人: [email protected]
主题: Re: ignite client memory issue
Hi,
Share piece of the code that produces the leak.
—
Denis
On Jan 22, 2017, at 5:56 PM, Shawn Du <[email protected]
<mailto:[email protected]> > wrote:
Hi,
My application run overnight and crash again after I set max heap as 2G. For I
saw there were many Future Objects,
I guess it may be caused by Async API. Now I am using Sync API. It seems that
the memory issue disappeared.
My application memory kept at 60M.
cache = cache.withAsync().withKeepBinary(); -->
cache = cache.withKeepBinary();
I think this will not happens always, but real happened on some condition and
worthy further investigate.
Thanks
Shawn
发件人: Shawn Du [mailto:[email protected]]
发送时间: 2017年1月22日 17:36
收件人: [email protected] <mailto:[email protected]>
主题: RE: ignite client memory issue
Hi,
I am sure there are memory leaks. See below.
Class Name
| Objects
| Shallow Heap | Retained Heap
---------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread
| 128 |
15,360 | >= 879,574,832
java.lang.ThreadLocal$ThreadLocalMap
| 100 | 2,400 | >=
870,478,984
java.lang.ThreadLocal$ThreadLocalMap$Entry[]
| 100 | 271,168 | >= 870,476,576
java.lang.ThreadLocal$ThreadLocalMap$Entry
| 23,153 | 740,896 | >= 868,443,568
org.apache.ignite.internal.processors.cache.IgniteCacheFutureImpl
| 22,444 | 359,104 | >= 867,056,800
org.apache.ignite.internal.util.future.GridFutureAdapter$ChainFuture
| 22,444 | 1,436,416 | >= 866,697,704
byte[]
| 26,362 | 864,837,368 | >= 864,837,368
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture|
22,441 | 3,411,032 | >= 861,188,352
org.apache.ignite.internal.binary.BinaryObjectImpl
| 22,441 | 897,640 | >= 855,099,616
---------------------------------------------------------------------------------------------------------------------------------------------
Now my application is still running, and the memory is growing up. Please help.
More information:
Ignite version: 1.8.0
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
Platform:
Linux dev-s2 4.4.8-20.46.amzn1.x86_64 #1 SMP Wed Apr 27 19:28:52 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux
This is a new issue in my application. Recently we change our code. Add below
Class which implement
public class Column<T> implements Binarylizable
{
}
Thanks. Please help.
Shawn
发件人: Shawn Du [ <mailto:[email protected]>
mailto:[email protected]]
发送时间: 2017年1月22日 13:52
收件人: <mailto:[email protected]> [email protected]
主题: Re: ignite client memory issue
Hi,
I assume this document for ignite server. In my case, I set ignite work at
client mode.
Now I increase the Max Heap Size and my application is running, I will monitor
the memory usage.
For my view, at client mode, ignite should not use too much memory.
Thanks
Shawn
发件人: Denis Magda [ <mailto:[email protected]> mailto:[email protected]]
发送时间: 2017年1月22日 13:38
收件人: <mailto:[email protected]> [email protected]
主题: Re: ignite client memory issue
Also keep in mind that every Ignite node requires at least ~ 300 MB for its
internal purposes. This capacity planning page might be useful as well for you:
<https://apacheignite.readme.io/docs/capacity-planning-bak>
https://apacheignite.readme.io/docs/capacity-planning-bak
—
Denis
On Jan 21, 2017, at 9:29 PM, Shawn Du < <mailto:[email protected]>
[email protected]> wrote:
My heap max size is 768M. more than 500M are consumed by ignite.
Your advice is quite reasonable. I will refactor my code.
Thanks
发件人: Denis Magda [ <mailto:[email protected]> mailto:[email protected]]
发送时间: 2017年1月22日 13:18
收件人: <mailto:[email protected]> [email protected]
主题: Re: ignite client memory issue
Hi Shawn,
What is the maximum size of the heap?
I don’t think the cache configurations can be a reason of the OOM. As a side
note, there is no reason to keep the configurations at all. Once a cache is
started with a configuration you can either keep a single reference to it and
reuse by multiple app threads or get a new one by passing the cache name into a
respective Ignite method.
—
Denis
On Jan 21, 2017, at 9:03 PM, Shawn Du < <mailto:[email protected]>
[email protected]> wrote:
I review the code, find a never released HashMap which store some cache
configurations.
private Map<TableSchema, CacheConfiguration<String, BinaryObject>>
cacheConfigurations = new HashMap<>();
I cache these configurations for performance consideration.
I get ignite cache by calling:
IgniteCache<String, BinaryObject> cache =
IgniteManager.getIgnite().getOrCreateCache(configuration);
My question:
If cache configuration doesn’t release, it will prevent some memeory/cache
entries/something else release?
Thanks
Shawn
发件人: Shawn Du [ <mailto:[email protected]>
mailto:[email protected]]
发送时间: 2017年1月22日 9:09
收件人: <mailto:[email protected]> [email protected]
主题: ignite client memory issue
Hi,
My ignite client died many times recently because of OOM. Of course, I can
increase the Max heap size. But I want to know why these memory are not
released.
This is part of analysis of Eclipse Memory Analyzer. My ignite client use Async
cache API. please help.
Class Name
| Objects | Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread
| 109 | 13,080 | >= 561,086,296
java.lang.ThreadLocal$ThreadLocalMap
| 98 | 2,352 | >= 537,825,592
java.lang.ThreadLocal$ThreadLocalMap$Entry[]
| 98 | 139,808 | >= 537,823,232
java.lang.ThreadLocal$ThreadLocalMap$Entry
| 15,202 | 486,464 | >= 535,750,616
org.apache.ignite.internal.processors.cache.IgniteCacheFutureImpl
| 14,476 | 231,616 | >= 534,866,208
org.apache.ignite.internal.util.future.GridFutureAdapter$ChainFuture
| 14,476 | 926,464 | >= 534,634,600
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture
| 14,466 | 2,198,832 | >= 531,444,584
byte[]
| 18,438 | 530,369,360 | >= 530,369,360
org.apache.ignite.internal.binary.BinaryObjectImpl
| 14,466 | 578,640 | >= 527,749,368
Thanks
Shawn
