I see these possible reasons of the delay 1. GC pause on client or server. Please enable GC logging and rerun test. 2. your machine runs out of resources - CPU, memory, disk IO, etc. Can you please check that there are enough resources in the system? 3. Is any energy saving stuff enabled?
I also don't think your benchmark is reasonable. You should properly warmup all participating JVMs to let them come to some steady state. For this I would run cache.get() from multiple threads without any sleeps just to make JVM to adapt, and only then start measuring the latency of cache operations. --Yakov 2016-07-19 12:46 GMT+03:00 tracel <[email protected]>: > sorry for late reply, really busy last week > > can reproduce the read delay with a standalone program now, and use > System.out.println() to time the cache read: > > public class App { > public static void main(String[] args) { > App app = new App(); > app.run(); > } > > private void run() { > readCache(); > } > > private void readCache() { > String cacheNamePrefix = "FOO"; > > TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi(); > TcpDiscoveryMulticastIpFinder ipFinder = new > TcpDiscoveryMulticastIpFinder(); > ipFinder.setMulticastGroup("228.10.11.166"); > discoverySpi.setIpFinder(ipFinder); > > IgniteConfiguration cfg = new IgniteConfiguration(); > cfg.setDiscoverySpi(discoverySpi); > > Ignition.setClientMode(true); > Ignite ignite = Ignition.start(cfg); > try { > String cacheName = cacheNamePrefix + "." + > Merchant.class.getName(); > IgniteCache<String, Merchant> cache = > ignite.getOrCreateCache(cacheName); > > readCache(cache); > } finally { > ignite.close(); > } > } > > private void readCache(IgniteCache<String, Merchant> cache) { > int sleepIntervalInMin = 3; > > while (sleepIntervalInMin < 30) { > try { > timeACacheRead(cache); > > Thread.sleep(1000 * 60 * > sleepIntervalInMin++); > } catch (InterruptedException e) { > e.printStackTrace(); > } > } > } > > private void timeACacheRead(IgniteCache<String, Merchant> > cache) { > SimpleDateFormat dateFormat = new > SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ"); > > System.out.println(dateFormat.format(new Date()) + " > before read"); > Merchant merchant = cache.get("BAR"); > System.out.println("merchant=" + merchant); > System.out.println(dateFormat.format(new Date()) + " after > read"); > } > > } > > The first 2 reads were good after idling less than 4 mins, > starting from the 3rd reading with idling time more than 4 mins, > most of the reads got 5 seconds delay (except the last one): > > [17:28:14] Topology snapshot [ver=6, servers=1, clients=1, CPUs=1, > heap=1.7GB] > 2016-07-19T17:28:14.284+0800 before read > log4j:WARN No appenders could be found for logger (org.jboss.logging). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig > for > more info. > merchant=com.foo.entity.Merchant@3feb2dda > 2016-07-19T17:28:14.938+0800 after read > 2016-07-19T17:31:14.938+0800 before read > merchant=com.foo.entity.Merchant@1c742ed4 > 2016-07-19T17:31:15.045+0800 after read > 2016-07-19T17:35:15.046+0800 before read > merchant=com.foo.entity.Merchant@55de24cc > 2016-07-19T17:35:20.109+0800 after read > 2016-07-19T17:40:20.109+0800 before read > merchant=com.foo.entity.Merchant@30f842ca > 2016-07-19T17:40:25.187+0800 after read > 2016-07-19T17:46:25.187+0800 before read > merchant=com.foo.entity.Merchant@4d14b6c2 > 2016-07-19T17:46:30.245+0800 after read > 2016-07-19T17:53:30.246+0800 before read > merchant=com.foo.entity.Merchant@c05fddc > 2016-07-19T17:53:30.340+0800 after read > > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/How-to-troubleshoot-a-slow-client-node-get-tp6250p6378.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
