Re: Ignite 2.x - can I take advantage of offheap storage and also still have time based eviction (from the on & off heap storage)?
This is the explanation I was looking for. I indeed was just not clear on
how these worked, but this helped me clear it up. thanks.
On Wed, Oct 21, 2020 at 11:44 PM Ilya Kazakov
wrote:
> Hello, Nicholas!
>
> As I understand from your code and from your explanation, it looks like
> you are a little confused about the Eviction Policies and the Expiry
> Policies.
>
> 1. Eviction policy determines which data should be removed from RAM if RAM
> runs out.
> 2. Expiry policy using for determining TTL of cache entries.
>
> In your case you need only Expiry policy. Look at my example:
>
> private static final String ORG_CACHE = IgniteCache.class.getSimpleName()
> + "Organizations";
>
> public static void main(String[] args) throws Exception {
> DataRegionConfiguration dfltDataRegConf = new
> DataRegionConfiguration();
> dfltDataRegConf.setPersistenceEnabled(true);
>
> DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
> dsCfg.setStoragePath("/home/kazakov/tmp");
>
> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> igniteConfiguration.setDataStorageConfiguration(dsCfg);
>
> TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
> tcpDiscoverySpi.setIpFinder(ipFinder);
>
> igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
>
> try(Ignite ignite = Ignition.start(igniteConfiguration)) {
> ignite.active(true);
>
> CacheConfiguration cacheCfg = new
> CacheConfiguration<>(ORG_CACHE);
>
> cacheCfg.setCacheMode(CacheMode.REPLICATED);
> cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheCfg.setBackups(1);
> cacheCfg.setOnheapCacheEnabled(true);
>
> IgniteCache cache =
> ignite.getOrCreateCache(cacheCfg).withExpiryPolicy(new
> CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 1)));
>
> for (long i = 0; i < 4_000_000; i++) {
> if (i > 0 && i % 10_000 == 0)
> System.out.println("Done: " + i);
>
> cache.put(i, new X12File("x12file" + i,
> LocalDateTime.now().toString()));
> }
>
> Thread.sleep(5000);
>
> int matches = 0;
> for (long i = 0; i < 4_000_000; i++) {
> if (cache.get(i) != null)
> ++matches;
> }
> System.out.println("Matches: " + matches);
> }
> }
>
> Ilya Kazakov
>
>
> ср, 21 окт. 2020 г. в 21:27, Nicholas DiPiazza <
> nicholas.dipia...@gmail.com>:
> >
> > Trying to get some help with my stack overflow question here
> https://stackoverflow.com/questions/64456578/apache-ignite-2-x-can-i-take-advantage-of-offheap-storage-and-also-still-have
> >
> > Cross posting it here.
> >
> > I am using Apache Ignite 2.8.1
> >
> > I have a specific cache that I need to store off heap (because otherwise
> I will be plagued with out of memory conditions). And I also need the cache
> to be distributed.
> >
> > But I also want a time based eviction policy to evict entries from
> on/off memory cache.
> >
> > Is this even possible with Apache Ignite 2.x?
> >
> > See the snippet here. I have tried to configure this every which way,
> but nothing evicts these entries from the cache except doing it manually.
> >
> > When I run the test method below, all the entries remain in the cache.
> >
> > import java.time.LocalDateTime;
> > import java.util.Arrays;
> > import java.util.concurrent.TimeUnit;
> >
> > import javax.cache.expiry.CreatedExpiryPolicy;
> > import javax.cache.expiry.Duration;
> >
> > import org.apache.ignite.Ignite;
> > import org.apache.ignite.Ignition;
> > import org.apache.ignite.cache.CacheAtomicityMode;
> > import org.apache.ignite.cache.CacheMode;
> > import org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory;
> > import org.apache.ignite.configuration.CacheConfiguration;
> > import org.apache.ignite.configuration.DataRegionConfiguration;
> > import org.apache.ignite.configuration.DataStorageConfiguration;
> > import org.apache.ignite.configuration.IgniteConfiguration;
> > import org.apache.ignite.configuration.NearCacheConfiguration;
> > import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> > import
> org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
> >
> > public class IgniteCache {
> > private static final String ORG_CACHE =
> IgniteCache.class.getSimpleName() + "Organizations";
> >
> > private static Ignite ignite;
> >
> > private static org.apache.ignite.IgniteCache cache;
> >
> > public void start() {
> >
> > IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
> > DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> > DataRegionConfiguration dfltDataRegConf = new
> DataRegionConfiguration();
> >
Re: Ignite 2.x - can I take advantage of offheap storage and also still have time based eviction (from the on & off heap storage)?
Hello, Nicholas!
As I understand from your code and from your explanation, it looks like you
are a little confused about the Eviction Policies and the Expiry Policies.
1. Eviction policy determines which data should be removed from RAM if RAM
runs out.
2. Expiry policy using for determining TTL of cache entries.
In your case you need only Expiry policy. Look at my example:
private static final String ORG_CACHE = IgniteCache.class.getSimpleName() +
"Organizations";
public static void main(String[] args) throws Exception {
DataRegionConfiguration dfltDataRegConf = new DataRegionConfiguration();
dfltDataRegConf.setPersistenceEnabled(true);
DataStorageConfiguration dsCfg = new DataStorageConfiguration();
dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
dsCfg.setStoragePath("/home/kazakov/tmp");
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setDataStorageConfiguration(dsCfg);
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
tcpDiscoverySpi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
try(Ignite ignite = Ignition.start(igniteConfiguration)) {
ignite.active(true);
CacheConfiguration cacheCfg = new
CacheConfiguration<>(ORG_CACHE);
cacheCfg.setCacheMode(CacheMode.REPLICATED);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setBackups(1);
cacheCfg.setOnheapCacheEnabled(true);
IgniteCache cache =
ignite.getOrCreateCache(cacheCfg).withExpiryPolicy(new
CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 1)));
for (long i = 0; i < 4_000_000; i++) {
if (i > 0 && i % 10_000 == 0)
System.out.println("Done: " + i);
cache.put(i, new X12File("x12file" + i,
LocalDateTime.now().toString()));
}
Thread.sleep(5000);
int matches = 0;
for (long i = 0; i < 4_000_000; i++) {
if (cache.get(i) != null)
++matches;
}
System.out.println("Matches: " + matches);
}
}
Ilya Kazakov
ср, 21 окт. 2020 г. в 21:27, Nicholas DiPiazza :
>
> Trying to get some help with my stack overflow question here
https://stackoverflow.com/questions/64456578/apache-ignite-2-x-can-i-take-advantage-of-offheap-storage-and-also-still-have
>
> Cross posting it here.
>
> I am using Apache Ignite 2.8.1
>
> I have a specific cache that I need to store off heap (because otherwise
I will be plagued with out of memory conditions). And I also need the cache
to be distributed.
>
> But I also want a time based eviction policy to evict entries from on/off
memory cache.
>
> Is this even possible with Apache Ignite 2.x?
>
> See the snippet here. I have tried to configure this every which way, but
nothing evicts these entries from the cache except doing it manually.
>
> When I run the test method below, all the entries remain in the cache.
>
> import java.time.LocalDateTime;
> import java.util.Arrays;
> import java.util.concurrent.TimeUnit;
>
> import javax.cache.expiry.CreatedExpiryPolicy;
> import javax.cache.expiry.Duration;
>
> import org.apache.ignite.Ignite;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheAtomicityMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.DataRegionConfiguration;
> import org.apache.ignite.configuration.DataStorageConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.configuration.NearCacheConfiguration;
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
>
> public class IgniteCache {
> private static final String ORG_CACHE =
IgniteCache.class.getSimpleName() + "Organizations";
>
> private static Ignite ignite;
>
> private static org.apache.ignite.IgniteCache cache;
>
> public void start() {
>
> IgniteConfiguration igniteConfiguration = new
IgniteConfiguration();
> DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> DataRegionConfiguration dfltDataRegConf = new
DataRegionConfiguration();
> dfltDataRegConf.setPersistenceEnabled(true);
> dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
> dsCfg.setStoragePath("E:\\igniteStorage");
> igniteConfiguration.setDataStorageConfiguration(dsCfg);
>
> TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
>
Re: [External]Re: Usage of TransactionConfiguration to overcome deadlocked threads
If you have user-defined types and can sort it by yourself, you can use LinkedHashMap as an argument to preserve the order and avoid deadlocks. ср, 21 окт. 2020 г. в 09:44, Kamlesh Joshi : > Thanks for the update Alex. > > > > Actually we are using BinaryObjects for such operations. Is there any > implementation available (or a reference) to sort user defined types of > objects ? > > > > *Thanks and Regards,* > > *Kamlesh Joshi* > > > > *From:* Alex Plehanov > *Sent:* 20 October 2020 19:54 > *To:* user@ignite.apache.org > *Subject:* [External]Re: Usage of TransactionConfiguration to overcome > deadlocked threads > > > > The e-mail below is from an external source. Please do not open > attachments or click links from an unknown or suspicious origin. > > Hello, > > > > TransactionConfiguration property has nothing to do with atomic caches. > Perhaps your threads were deadlocked due to atomic putAll/removeAll > operations with an unordered set of keys. It's a known issue and I hope > will be fixed soon. See [1] for detailed information. Until this ticked is > fixed you should avoid concurrent putAll/removeAll operations with an > unordered set of keys on atomic caches (putAll with HashMap as an argument, > for example). > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-12451 > > > > вт, 20 окт. 2020 г. в 12:16, Kamlesh Joshi : > > Hi Igniters, > > > > We are currently using ATOMIC caches for our operations. Recently, we > observed cluster hang issue, the operations were stuck for quite a long > time (had to bring down the cluster to resolve this). So, after some > digging found that setting up below property should resolve this. Could you > please confirm on below: > > > >1. Whether this needs to be set on both Ignite servers and Ignite >thick clients? >2. Or setting on cluster should suffice? >3. What should be the optimum value for *defaultTxTimeout* > > > > > > ** > > * class="org.apache.ignite.configuration.TransactionConfiguration">* > > ** > > ** > > ** > > > > > > > > > > *Thanks and Regards,* > > *Kamlesh Joshi* > > > > > "*Confidentiality Warning*: This message and any attachments are intended > only for the use of the intended recipient(s), are confidential and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, re-transmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return email and delete this message and any attachments > from your system. > > *Virus Warning:* Although the company has taken reasonable precautions to > ensure no viruses are present in this email. The company cannot accept > responsibility for any loss or damage arising from the use of this email or > attachment." > > > "*Confidentiality Warning*: This message and any attachments are intended > only for the use of the intended recipient(s), are confidential and may be > privileged. If you are not the intended recipient, you are hereby notified > that any review, re-transmission, conversion to hard copy, copying, > circulation or other use of this message and any attachments is strictly > prohibited. If you are not the intended recipient, please notify the sender > immediately by return email and delete this message and any attachments > from your system. > > *Virus Warning:* Although the company has taken reasonable precautions to > ensure no viruses are present in this email. The company cannot accept > responsibility for any loss or damage arising from the use of this email or > attachment." >
Ignite 2.x - can I take advantage of offheap storage and also still have time based eviction (from the on & off heap storage)?
Trying to get some help with my stack overflow question here
https://stackoverflow.com/questions/64456578/apache-ignite-2-x-can-i-take-advantage-of-offheap-storage-and-also-still-have
Cross posting it here.
I am using Apache Ignite 2.8.1
I have a specific cache that I need to store off heap (because otherwise I
will be plagued with out of memory conditions). And I also need the cache
to be distributed.
But I also want a time based eviction policy to evict entries from on/off
memory cache.
Is this even possible with Apache Ignite 2.x?
See the snippet here. I have tried to configure this every which way, but
nothing evicts these entries from the cache except doing it manually.
When I run the test method below, all the entries remain in the cache.
import java.time.LocalDateTime;import java.util.Arrays;import
java.util.concurrent.TimeUnit;
import javax.cache.expiry.CreatedExpiryPolicy;import
javax.cache.expiry.Duration;
import org.apache.ignite.Ignite;import
org.apache.ignite.Ignition;import
org.apache.ignite.cache.CacheAtomicityMode;import
org.apache.ignite.cache.CacheMode;import
org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory;import
org.apache.ignite.configuration.CacheConfiguration;import
org.apache.ignite.configuration.DataRegionConfiguration;import
org.apache.ignite.configuration.DataStorageConfiguration;import
org.apache.ignite.configuration.IgniteConfiguration;import
org.apache.ignite.configuration.NearCacheConfiguration;import
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
public class IgniteCache {
private static final String ORG_CACHE =
IgniteCache.class.getSimpleName() + "Organizations";
private static Ignite ignite;
private static org.apache.ignite.IgniteCache cache;
public void start() {
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
DataStorageConfiguration dsCfg = new DataStorageConfiguration();
DataRegionConfiguration dfltDataRegConf = new DataRegionConfiguration();
dfltDataRegConf.setPersistenceEnabled(true);
dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
dsCfg.setStoragePath("E:\\igniteStorage");
igniteConfiguration.setDataStorageConfiguration(dsCfg);
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
tcpDiscoverySpi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
ignite = Ignition.start(igniteConfiguration);
ignite.active(true);
CacheConfiguration cacheCfg = new
CacheConfiguration<>(ORG_CACHE);
cacheCfg.setCacheMode(CacheMode.REPLICATED);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setBackups(1);
cacheCfg.setEvictionPolicyFactory(new
LruEvictionPolicyFactory<>(5, 40, 5050)); // this doesn't do
anything.
cacheCfg.setNearConfiguration(
new NearCacheConfiguration()
.setNearEvictionPolicyFactory(new
LruEvictionPolicyFactory<>(1_000_000))); // this doesn't do anything.
cache = ignite.getOrCreateCache(cacheCfg);
for (long i = 0; i < 4_000_000; i++) {
if (i > 0 && i % 10_000 == 0) {
System.out.println("Done: " + i);
}
cache.withExpiryPolicy(new CreatedExpiryPolicy(new
Duration(TimeUnit.SECONDS, 1))) // this expiry policy doesn't do
anything
.put(i, new X12File("x12file" + i,
LocalDateTime.now().toString()));
}
}
public void test() {
System.out.println("Checking if cache entries are being
properly evicted ...");
int matches = 0;
for (long i = 0; i < 4_000_000; i++) {
if (cache.get(i) != null) {
++matches;
}
}
System.out.println("Matches: " + matches);
}
}
Is this just a shortcoming of Apache Ignite 2.x in my particular use case?
RE: [External]Re: Usage of TransactionConfiguration to overcome deadlocked threads
Thanks for the update Alex. Actually we are using BinaryObjects for such operations. Is there any implementation available (or a reference) to sort user defined types of objects ? Thanks and Regards, Kamlesh Joshi From: Alex Plehanov Sent: 20 October 2020 19:54 To: user@ignite.apache.org Subject: [External]Re: Usage of TransactionConfiguration to overcome deadlocked threads The e-mail below is from an external source. Please do not open attachments or click links from an unknown or suspicious origin. Hello, TransactionConfiguration property has nothing to do with atomic caches. Perhaps your threads were deadlocked due to atomic putAll/removeAll operations with an unordered set of keys. It's a known issue and I hope will be fixed soon. See [1] for detailed information. Until this ticked is fixed you should avoid concurrent putAll/removeAll operations with an unordered set of keys on atomic caches (putAll with HashMap as an argument, for example). [1]: https://issues.apache.org/jira/browse/IGNITE-12451 вт, 20 окт. 2020 г. в 12:16, Kamlesh Joshi mailto:kamlesh.jo...@ril.com>>: Hi Igniters, We are currently using ATOMIC caches for our operations. Recently, we observed cluster hang issue, the operations were stuck for quite a long time (had to bring down the cluster to resolve this). So, after some digging found that setting up below property should resolve this. Could you please confirm on below: 1. Whether this needs to be set on both Ignite servers and Ignite thick clients? 2. Or setting on cluster should suffice? 3. What should be the optimum value for defaultTxTimeout Thanks and Regards, Kamlesh Joshi "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential and may be privileged. If you are not the intended recipient, you are hereby notified that any review, re-transmission, conversion to hard copy, copying, circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment." "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
