Ignite Cluster: Cache Misses happening for existing keys

[HEWA WIDANA GAMAGE, SUBASH]

Hi all,
We're observing this in a 3 node server cluster(3 separate JVMs, in 3 separate 
VMs within same datacenter, node to node latency is 2-3 milliseconds within the 
network).

As with following code is wrapped inside an http API, and that API is getting 
called by a 5000 users ramping up(60seconds) forever, hitting 3 nodes in a 
round robin manner.

With this, for same cache key, I could see more than one "Cache put key=" log 
appearing with in 15mins window(actually I am getting this duplicate put logs 
after 2-3mins of the load test)..

For the SAME cache key, there cannot be more than one put within 15mins. Based 
on cache size, it's well below eviction size, and since it's well within the 
expiry window, looks to me some timing issue when replicating the cache between 
the nodes.

Time between same key cache put logs is about 8-12 seconds usually.  Am I doing 
something wrong here ? Any way we can make a cache.put operation synchronously 
complete only upon a full node replication (not quite sure whether it helps 
though)?

Version: Ignite 1.9

Code to create the cache

IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDiscoverySpi(getDiscoverySpi()); // static ip list on tcp discovery
cfg.setClientMode(false);
cfg.setIncludeEventTypes(EventType.EVT_NODE_SEGMENTED,EventType.EVT_NODE_FAILED);
Ignite ignite = Ignition.start(cfg);

ignite.events().localListen(event -> {
    LOG.info("Cache event received: {} ", event);
    return true;},EventType.EVT_NODE_SEGMENTED, EventType.EVT_NODE_FAILED);


CacheConfiguration<String,String> cc  = new CacheConfiguration<>();
cc.setName("mycache1");
cc.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new 
Duration(TimeUnit.MINUTES, 15)));
cc.setCacheMode(CacheMode.REPLICATED);
  LruEvictionPolicy evictionPolicy = new LruEvictionPolicy();
  evictionPolicy.setMaxMemorySize(500 * 1024 * 1024);
cc.setEvictionPolicy(evictionPolicy);

IgniteCache<String,String> cache = ignite.getOrCreateCache(cc);


Code to cache operations.(following method can be accessed by multiple threads 
at the same time)


private static String processKey(String key) {
    String value = cache.get(key);
    if (value == null) {
        cache.put(key, getNewValue());
        LOG.info("Cache put key={} ", key);
    } else {
        LOG.info("Cache hit key={}", key);
    }
    return value;
}