RE: Ignite Cluster: Cache Misses happening for existing keys
With your idea of that race condition, we decided to lock the get, and put for
a given key on app level, and keep using the JCache API. So far for 250 tps
with 3 nodes, everything looks good.
Thank you very much!
From: Вячеслав Коптилин [mailto:slava.kopti...@gmail.com]
Sent: Thursday, September 13, 2018 9:01 AM
To: user@ignite.apache.org
Subject: Re: Ignite Cluster: Cache Misses happening for existing keys
This email is from an external source - exercise caution regarding links and
attachments. Please visit cybersecurity.fmr.com/phishing/ to learn more.
Hi,
You can try with using EntryProcessor. Please take a look at the following
example:
public class EntryUpdater implements EntryProcessor {
@Override
public String process(MutableEntry entry, Object...
arguments) throws EntryProcessorException {
if (!entry.exists()) {
// Cache does not contain a mapping for the given key.
String newValue = getNewValue();
entry.setValue(newValue);
}
else {
// The entry is already exists, just return the existing value.
}
return entry.getValue();
}
}
private static String processKey(String key) {
return cache.invoke(key, new EntryUpdater());
}
[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#invoke-K-org.apache.ignite.cache.CacheEntryProcessor-java.lang.Object...-
Thanks,
S.
чт, 13 сент. 2018 г. в 15:06, HEWA WIDANA GAMAGE, SUBASH
mailto:subash.hewawidanagam...@fmr.com>>:
Thank you for the prompt response.
I agree there’s a race condition here. But the problem of the suggestion is,
“getNewValue()” will get called even for cache hits. This getNewValue
eventually calling some backend legacy API and the caching layer is setup to
reduce the load for it.
From: Вячеслав Коптилин
[mailto:slava.kopti...@gmail.com<mailto:slava.kopti...@gmail.com>]
Sent: Thursday, September 13, 2018 4:08 AM
To: user@ignite.apache.org<mailto:user@ignite.apache.org>
Subject: Re: Ignite Cluster: Cache Misses happening for existing keys
This email is from an external source - exercise caution regarding links and
attachments. Please visit
cybersecurity.fmr.com/phishing/<http://cybersecurity.fmr.com/phishing/> to
learn more.
Hello,
Your code contains the following obvious race:
String value = cache.get(key);
if (value == null) {
cache.put(key, getNewValue());
LOG.info("Cache put key={} ", key);
}
It's easy to imagine that two threads managed to execute get() method and got
null value before put() operation is called.
Please try to modify the code as follows:
if (cache.putIfAbsent(key, getNewValue())) {
LOG.info("Cache put key={} ", key);
}
else {
LOG.info("Cache hit key={}", key);
}
[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#putIfAbsent-K-V-
Thanks,
S.
чт, 13 сент. 2018 г. в 6:29, HEWA WIDANA GAMAGE, SUBASH
mailto:subash.hewawidanagam...@fmr.com>>:
Hi all,
We’re observing this in a 3 node server cluster(3 separate JVMs, in 3 separate
VMs within same datacenter, node to node latency is 2-3 milliseconds within the
network).
As with following code is wrapped inside an http API, and that API is getting
called by a 5000 users ramping up(60seconds) forever, hitting 3 nodes in a
round robin manner.
With this, for same cache key, I could see more than one “Cache put key=” log
appearing with in 15mins window(actually I am getting this duplicate put logs
after 2-3mins of the load test)..
For the SAME cache key, there cannot be more than one put within 15mins. Based
on cache size, it’s well below eviction size, and since it’s well within the
expiry window, looks to me some timing issue when replicating the cache between
the nodes.
Time between same key cache put logs is about 8-12 seconds usually. Am I doing
something wrong here ? Any way we can make a cache.put operation synchronously
complete only upon a full node replication (not quite sure whether it helps
though)?
Version: Ignite 1.9
Code to create the cache
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDiscoverySpi(getDiscoverySpi()); // static ip list on tcp discovery
cfg.setClientMode(false);
cfg.setIncludeEventTypes(EventType.EVT_NODE_SEGMENTED,EventType.EVT_NODE_FAILED);
Ignite ignite = Ignition.start(cfg);
ignite.events().localListen(event -> {
LOG.info("Cache event received: {} ", event);
return true;},EventType.EVT_NODE_SEGMENTED, EventType.EVT_NODE_FAILED);
CacheConfiguration cc = new CacheConfiguration<>();
cc.setName("mycache1");
cc.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
Duration(TimeUnit.MINUTES, 15)));
cc.setCacheMode(CacheMode.REPLICATED);
LruEvictionPolicy evictionPolicy = new LruEvictionPolicy();
evictionPolicy.setMaxMemorySize(500 * 1024 *
Re: Ignite Cluster: Cache Misses happening for existing keys
Hi,
You can try with using EntryProcessor. Please take a look at the following
example:
public class EntryUpdater implements EntryProcessor {
@Override
public String process(MutableEntry entry,
Object... arguments) throws EntryProcessorException {
if (!entry.exists()) {
// Cache does not contain a mapping for the given key.
String newValue = getNewValue();
entry.setValue(newValue);
}
else {
// The entry is already exists, just return the existing value.
}
return entry.getValue();
}
}
private static String processKey(String key) {
return cache.invoke(key, new EntryUpdater());
}
[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#invoke-K-org.apache.ignite.cache.CacheEntryProcessor-java.lang.Object...-
Thanks,
S.
чт, 13 сент. 2018 г. в 15:06, HEWA WIDANA GAMAGE, SUBASH <
subash.hewawidanagam...@fmr.com>:
> Thank you for the prompt response.
>
>
>
> I agree there’s a race condition here. But the problem of the suggestion
> is, “getNewValue()” will get called even for cache hits. This getNewValue
> eventually calling some backend legacy API and the caching layer is setup
> to reduce the load for it.
>
>
>
> *From:* Вячеслав Коптилин [mailto:slava.kopti...@gmail.com]
> *Sent:* Thursday, September 13, 2018 4:08 AM
> *To:* user@ignite.apache.org
> *Subject:* Re: Ignite Cluster: Cache Misses happening for existing keys
>
>
> This email is from an external source - exercise caution regarding links
> and attachments. Please visit cybersecurity.fmr.com/phishing/ to learn
> more.
>
>
>
> Hello,
>
>
>
> Your code contains the following obvious race:
>
> String value = *cache*.get(key);
> *if *(value == *null*) {
> *cache*.put(key, *getNewValue*());
> *LOG*.info(*"Cache put key={} "*, key);
> }
>
> It's easy to imagine that two threads managed to execute get() method and got
> null value before put() operation is called.
>
> Please try to modify the code as follows:
>
> if (cache.putIfAbsent(key, getNewValue())) {
>
> LOG.info("Cache put key={} ", key);
>
> }
>
> else {
>
> LOG.info("Cache hit key={}", key);
>
> }
>
>
>
> [1]
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#putIfAbsent-K-V-
>
>
>
> Thanks,
>
> S.
>
>
>
> чт, 13 сент. 2018 г. в 6:29, HEWA WIDANA GAMAGE, SUBASH <
> subash.hewawidanagam...@fmr.com>:
>
> Hi all,
>
> We’re observing this in a 3 node server cluster(3 separate JVMs, in 3
> separate VMs within same datacenter, node to node latency is 2-3
> milliseconds within the network).
>
>
>
> As with following code is wrapped inside an http API, and that API is
> getting called by a 5000 users ramping up(60seconds) forever, hitting 3
> nodes in a round robin manner.
>
>
>
> *With this, for same cache key, I could see more than one “Cache put key=”
> log appearing with in 15mins window(actually I am getting this duplicate
> put logs after 2-3mins of the load test).. *
>
>
>
> For the SAME cache key, there cannot be more than one put within 15mins.
> Based on cache size, it’s well below eviction size, and since it’s well
> within the expiry window, looks to me some timing issue when replicating
> the cache between the nodes.
>
>
>
> Time between same key cache put logs is about 8-12 seconds usually. Am I
> doing something wrong here ? Any way we can make a cache.put operation
> synchronously complete only upon a full node replication (not quite sure
> whether it helps though)?
>
>
>
> Version: Ignite 1.9
>
>
>
> *Code to create the cache*
>
>
>
> IgniteConfiguration cfg = *new *IgniteConfiguration();
> cfg.setDiscoverySpi(*getDiscoverySpi*());
> *// static ip list on tcp discovery *cfg.setClientMode(*false*);
> cfg.setIncludeEventTypes(EventType.*EVT_NODE_SEGMENTED*,EventType.
> *EVT_NODE_FAILED*);
> Ignite ignite = Ignition.*start*(cfg);
>
> ignite.events().localListen(event -> {
> *LOG*.info(*"Cache event received: {} "*, event);
> *return true*;},EventType.*EVT_NODE_SEGMENTED*,
> EventType.*EVT_NODE_FAILED*);
>
>
>
> CacheConfiguration cc = *new *CacheConfiguration<>();
> cc.setName(*"mycache1"*);
> cc.setExpiryPolicyFactory(CreatedExpiryPolicy.*factoryOf*(*new
> *Duration(TimeUnit.*MINUTES*, 15)));
> cc.setCacheMode(CacheMode.*REPLICATED*);
> LruEvictionPolicy evictionPolicy = *new *LruEvictionPolicy();
> evictionPolicy.setMaxMemorySize(500 * 1024 * 1024);
> cc.setEviction
RE: Ignite Cluster: Cache Misses happening for existing keys
Thank you for the prompt response.
I agree there’s a race condition here. But the problem of the suggestion is,
“getNewValue()” will get called even for cache hits. This getNewValue
eventually calling some backend legacy API and the caching layer is setup to
reduce the load for it.
From: Вячеслав Коптилин [mailto:slava.kopti...@gmail.com]
Sent: Thursday, September 13, 2018 4:08 AM
To: user@ignite.apache.org
Subject: Re: Ignite Cluster: Cache Misses happening for existing keys
This email is from an external source - exercise caution regarding links and
attachments. Please visit cybersecurity.fmr.com/phishing/ to learn more.
Hello,
Your code contains the following obvious race:
String value = cache.get(key);
if (value == null) {
cache.put(key, getNewValue());
LOG.info("Cache put key={} ", key);
}
It's easy to imagine that two threads managed to execute get() method and got
null value before put() operation is called.
Please try to modify the code as follows:
if (cache.putIfAbsent(key, getNewValue())) {
LOG.info("Cache put key={} ", key);
}
else {
LOG.info("Cache hit key={}", key);
}
[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#putIfAbsent-K-V-
Thanks,
S.
чт, 13 сент. 2018 г. в 6:29, HEWA WIDANA GAMAGE, SUBASH
mailto:subash.hewawidanagam...@fmr.com>>:
Hi all,
We’re observing this in a 3 node server cluster(3 separate JVMs, in 3 separate
VMs within same datacenter, node to node latency is 2-3 milliseconds within the
network).
As with following code is wrapped inside an http API, and that API is getting
called by a 5000 users ramping up(60seconds) forever, hitting 3 nodes in a
round robin manner.
With this, for same cache key, I could see more than one “Cache put key=” log
appearing with in 15mins window(actually I am getting this duplicate put logs
after 2-3mins of the load test)..
For the SAME cache key, there cannot be more than one put within 15mins. Based
on cache size, it’s well below eviction size, and since it’s well within the
expiry window, looks to me some timing issue when replicating the cache between
the nodes.
Time between same key cache put logs is about 8-12 seconds usually. Am I doing
something wrong here ? Any way we can make a cache.put operation synchronously
complete only upon a full node replication (not quite sure whether it helps
though)?
Version: Ignite 1.9
Code to create the cache
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDiscoverySpi(getDiscoverySpi()); // static ip list on tcp discovery
cfg.setClientMode(false);
cfg.setIncludeEventTypes(EventType.EVT_NODE_SEGMENTED,EventType.EVT_NODE_FAILED);
Ignite ignite = Ignition.start(cfg);
ignite.events().localListen(event -> {
LOG.info("Cache event received: {} ", event);
return true;},EventType.EVT_NODE_SEGMENTED, EventType.EVT_NODE_FAILED);
CacheConfiguration cc = new CacheConfiguration<>();
cc.setName("mycache1");
cc.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
Duration(TimeUnit.MINUTES, 15)));
cc.setCacheMode(CacheMode.REPLICATED);
LruEvictionPolicy evictionPolicy = new LruEvictionPolicy();
evictionPolicy.setMaxMemorySize(500 * 1024 * 1024);
cc.setEvictionPolicy(evictionPolicy);
IgniteCache cache = ignite.getOrCreateCache(cc);
Code to cache operations.(following method can be accessed by multiple threads
at the same time)
private static String processKey(String key) {
String value = cache.get(key);
if (value == null) {
cache.put(key, getNewValue());
LOG.info("Cache put key={} ", key);
} else {
LOG.info("Cache hit key={}", key);
}
return value;
}
Re: Ignite Cluster: Cache Misses happening for existing keys
Hello,
Your code contains the following obvious race:
String value = *cache*.get(key);
*if *(value == *null*) {
*cache*.put(key, *getNewValue*());
*LOG*.info(*"Cache put key={} "*, key);
}
It's easy to imagine that two threads managed to execute get() method
and got null value before put() operation is called.
Please try to modify the code as follows:
if (cache.putIfAbsent(key, getNewValue())) {
LOG.info("Cache put key={} ", key);
}
else {
LOG.info("Cache hit key={}", key);
}
[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/IgniteCache.html#putIfAbsent-K-V-
Thanks,
S.
чт, 13 сент. 2018 г. в 6:29, HEWA WIDANA GAMAGE, SUBASH <
subash.hewawidanagam...@fmr.com>:
> Hi all,
>
> We’re observing this in a 3 node server cluster(3 separate JVMs, in 3
> separate VMs within same datacenter, node to node latency is 2-3
> milliseconds within the network).
>
>
>
> As with following code is wrapped inside an http API, and that API is
> getting called by a 5000 users ramping up(60seconds) forever, hitting 3
> nodes in a round robin manner.
>
>
>
> *With this, for same cache key, I could see more than one “Cache put key=”
> log appearing with in 15mins window(actually I am getting this duplicate
> put logs after 2-3mins of the load test).. *
>
>
>
> For the SAME cache key, there cannot be more than one put within 15mins.
> Based on cache size, it’s well below eviction size, and since it’s well
> within the expiry window, looks to me some timing issue when replicating
> the cache between the nodes.
>
>
>
> Time between same key cache put logs is about 8-12 seconds usually. Am I
> doing something wrong here ? Any way we can make a cache.put operation
> synchronously complete only upon a full node replication (not quite sure
> whether it helps though)?
>
>
>
> Version: Ignite 1.9
>
>
>
> *Code to create the cache*
>
>
>
> IgniteConfiguration cfg = *new *IgniteConfiguration();
> cfg.setDiscoverySpi(*getDiscoverySpi*());
> *// static ip list on tcp discovery *cfg.setClientMode(*false*);
> cfg.setIncludeEventTypes(EventType.*EVT_NODE_SEGMENTED*,EventType.
> *EVT_NODE_FAILED*);
> Ignite ignite = Ignition.*start*(cfg);
>
> ignite.events().localListen(event -> {
> *LOG*.info(*"Cache event received: {} "*, event);
> *return true*;},EventType.*EVT_NODE_SEGMENTED*,
> EventType.*EVT_NODE_FAILED*);
>
>
>
> CacheConfiguration cc = *new *CacheConfiguration<>();
> cc.setName(*"mycache1"*);
> cc.setExpiryPolicyFactory(CreatedExpiryPolicy.*factoryOf*(*new
> *Duration(TimeUnit.*MINUTES*, 15)));
> cc.setCacheMode(CacheMode.*REPLICATED*);
> LruEvictionPolicy evictionPolicy = *new *LruEvictionPolicy();
> evictionPolicy.setMaxMemorySize(500 * 1024 * 1024);
> cc.setEvictionPolicy(evictionPolicy);
>
>
> IgniteCache cache = ignite.getOrCreateCache(cc);
>
>
>
>
>
> *Code to cache operations.(following method can be accessed by multiple
> threads at the same time)*
>
>
>
> *private static *String processKey(String key) {
> String value = *cache*.get(key);
> *if *(value == *null*) {
> *cache*.put(key, *getNewValue*());
> *LOG*.info(*"Cache put key={} "*, key);
> } *else *{
> *LOG*.info(*"Cache hit key={}"*, key);
> }
> *return *value;
> }
>
>
>
>
>
>
>
