Re: eviction performance
Hi Scott, With eagetTTL=false, each time you access an entry its TTL is automatically checked - if it is expired, the entry will be removed. It means that instead of having a separate thread waking up and removing all of the expired entries at the same time (and taking a chunk of CPU and IO for a noticeable period of time), you'll be removing them over time. On average, individual accesses may become a bit more expensive (because some of them will trigger removal) but that will allow you to avoid latency spikes. About the truncate - I think there is no API to do that based on partitions. I guess you could try creating a new cache for each batch and destroying it after you don't need it anymore, but it's hard to say whether it will be more efficient. Stan -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: eviction performance
Hi Stanislav,
Thanks for info and the note on the terminology.
So in my setup im partitioning by chunks of time. If I turn eagerttl to false
how would the cleanup look? Would I periodically scan the partitions that i
want to be expired? Are there any best practices to that end?
Additionally are there any truncate-like commands I can use to achieve a
lightweight cleanup of my data?
Thanks,
Scott
> On Mar 9, 2018, at 1:41 AM, Stanislav Lukyanov <stanlukya...@gmail.com> wrote:
>
> Hi,
>
> A terminology nitpicking: it seems that you’re talking about expiry; eviction
> is a similar mechanism but is based on the data size, not time.
>
> Have you tried setting CacheConfiguration.eagerTtl = false
> (https://apacheignite.readme.io/v2.3/docs/expiry-policies#section-eager-ttl)?
> With that setting the expired entries will be removed on the next access
> instead of doing that in the background.
> If you add entries in large batches but access them more uniformly over time,
> it could redistribute the cost of expiring the data more evenly.
>
> Thanks,
> Stan
>
> From: scottmf
> Sent: 9 марта 2018 г. 6:51
> To: user@ignite.apache.org
> Subject: eviction performance
>
> Hi,
> I am prototyping using ignite to ingest lots of short lived events in a
> system. In order to ingest the data i'm using the kafka streamer mechanism.
> I'm pushing 20k events / second into the system into a partitioned off heap
> cache. I've run into a few performance issues but I've been able to wiggle
> my way out of it using information from the forums and ignite docs. But the
> latest issue I can't seem to find any information about.
>
> My cache is setup to evict the data after one hour. The insertion
> performance degrades substantially when the eviction mechanism kicks in
> where the latency of the insertion continues to degrade over time.
>
> WRT to the OS, memory and disk none of them are saturated. The load avg is
> low, the gc is not under pressure and the disk is not fully utilized.
>
> I've experimented around with different eviction times to make sure that it
> is indeed the eviction that is causing this. After some idle time I startup
> our event simulator and wait for the evictions to start and sure enough the
> latency of insertions increases immediately when evictions start.
>
> Each node is 32 GB / 8 cpus.
>
> Here are my jvm opts: "-Xmx12g -Xms12g -XX:MaxDirectMemorySize=10g
> -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseG1GC -XX:ConcGCThreads=12
> -XX:ParallelGCThreads=22 -XX:MaxGCPauseMillis=1000 -XX:G1HeapWastePercent=2
> -XX:G1ReservePercent=15 -XX:+UnlockExperimentalVMOptions
> -XX:G1OldCSetRegionThresholdPercent=15 -XX:G1MixedGCLiveThresholdPercent=90
> -XX:G1MaxNewSizePercent=25 -Dcom.sun.management.jmxremote
> -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
>
> I was wondering if there is no tuning I can do to avoid this overhead, I am
> partitioning my data by time - is there a performant way to clear a
> partition in a cache one at a time? (i was thinking of something similar to
> a truncate command in a rdbms)
>
> Here is my schema:
> CachePojo {
> @QuerySqlField(index=false)
> private String buffer0;
> @QuerySqlField(index=true)
> private String buffer1;
> @QuerySqlField
> private String buffer2;
> @QuerySqlField(index=true, inlineSize=8, descending=true)
> private Timestamp timestamp;
> @QuerySqlField(index=false)
> private Map<String, String> fields = new HashMap<>();
> }
>
> Ignite config:
> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> igniteConfiguration.setIgniteHome(igniteHome);
> // this is probably not needed
> igniteConfiguration.setTransactionConfiguration(new
> TransactionConfiguration());
> igniteConfiguration.setPeerClassLoadingEnabled(true);
> igniteConfiguration.setIncludeEventTypes(new int[0]);
> igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi());
> igniteConfiguration.setDataStreamerThreadPoolSize(8);
> igniteConfiguration.setPublicThreadPoolSize(16);
> igniteConfiguration.setSystemThreadPoolSize(16);
> DataStorageConfiguration dsc = dataStorageConfiguration();
> igniteConfiguration.setDataStorageConfiguration(dsc);
>
> dataStorageConfig:
> DataStorageConfiguration dataStorageConfiguration = new
> DataStorageConfiguration();
> DataRegionConfiguration dataRegionConfiguration =
> dataRegionConfiguration();
>
> dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
>
> dataStorageConfiguration.setDataRe
RE: eviction performance
Hi,
A terminology nitpicking: it seems that you’re talking about expiry; eviction
is a similar mechanism but is based on the data size, not time.
Have you tried setting CacheConfiguration.eagerTtl = false
(https://apacheignite.readme.io/v2.3/docs/expiry-policies#section-eager-ttl)?
With that setting the expired entries will be removed on the next access
instead of doing that in the background.
If you add entries in large batches but access them more uniformly over time,
it could redistribute the cost of expiring the data more evenly.
Thanks,
Stan
From: scottmf
Sent: 9 марта 2018 г. 6:51
To: user@ignite.apache.org
Subject: eviction performance
Hi,
I am prototyping using ignite to ingest lots of short lived events in a
system. In order to ingest the data i'm using the kafka streamer mechanism.
I'm pushing 20k events / second into the system into a partitioned off heap
cache. I've run into a few performance issues but I've been able to wiggle
my way out of it using information from the forums and ignite docs. But the
latest issue I can't seem to find any information about.
My cache is setup to evict the data after one hour. The insertion
performance degrades substantially when the eviction mechanism kicks in
where the latency of the insertion continues to degrade over time.
WRT to the OS, memory and disk none of them are saturated. The load avg is
low, the gc is not under pressure and the disk is not fully utilized.
I've experimented around with different eviction times to make sure that it
is indeed the eviction that is causing this. After some idle time I startup
our event simulator and wait for the evictions to start and sure enough the
latency of insertions increases immediately when evictions start.
Each node is 32 GB / 8 cpus.
Here are my jvm opts: "-Xmx12g -Xms12g -XX:MaxDirectMemorySize=10g
-XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseG1GC -XX:ConcGCThreads=12
-XX:ParallelGCThreads=22 -XX:MaxGCPauseMillis=1000 -XX:G1HeapWastePercent=2
-XX:G1ReservePercent=15 -XX:+UnlockExperimentalVMOptions
-XX:G1OldCSetRegionThresholdPercent=15 -XX:G1MixedGCLiveThresholdPercent=90
-XX:G1MaxNewSizePercent=25 -Dcom.sun.management.jmxremote
-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
I was wondering if there is no tuning I can do to avoid this overhead, I am
partitioning my data by time - is there a performant way to clear a
partition in a cache one at a time? (i was thinking of something similar to
a truncate command in a rdbms)
Here is my schema:
CachePojo {
@QuerySqlField(index=false)
private String buffer0;
@QuerySqlField(index=true)
private String buffer1;
@QuerySqlField
private String buffer2;
@QuerySqlField(index=true, inlineSize=8, descending=true)
private Timestamp timestamp;
@QuerySqlField(index=false)
private Map<String, String> fields = new HashMap<>();
}
Ignite config:
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setIgniteHome(igniteHome);
// this is probably not needed
igniteConfiguration.setTransactionConfiguration(new
TransactionConfiguration());
igniteConfiguration.setPeerClassLoadingEnabled(true);
igniteConfiguration.setIncludeEventTypes(new int[0]);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi());
igniteConfiguration.setDataStreamerThreadPoolSize(8);
igniteConfiguration.setPublicThreadPoolSize(16);
igniteConfiguration.setSystemThreadPoolSize(16);
DataStorageConfiguration dsc = dataStorageConfiguration();
igniteConfiguration.setDataStorageConfiguration(dsc);
dataStorageConfig:
DataStorageConfiguration dataStorageConfiguration = new
DataStorageConfiguration();
DataRegionConfiguration dataRegionConfiguration =
dataRegionConfiguration();
dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
dataStorageConfiguration.setDataRegionConfigurations(largeDataRegionConfiguration());
dataStorageConfiguration.setPageSize(8192);
dataStorageConfiguration.setMetricsEnabled(true);
dataStorageConfiguration.setWriteThrottlingEnabled(true);
dataStorageConfiguration.setStoragePath("/var/lib/ignite/persistence");
dataStorageConfiguration.setWalMode(WALMode.NONE);
dataStorageConfiguration.setWalPath("/var/lib/ignite/wal");
dataStorageConfiguration.setWalArchivePath("/var/lib/ignite/wal/archive");
cacheDataRegionConfig:
DataRegionConfiguration dataRegionConfiguration = new
DataRegionConfiguration();
dataRegionConfiguration.setPersistenceEnabled(true);
dataRegionConfiguration.setName("dataRegion");
dataRegionConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
dataRegionConfiguration.setInitialSize(500l * 1024 * 1024);
