Hi! Sorry for the long question in advance...
I'm doing some performance measurements on Ignite to see if it would work for a new project. So far, I love Ignite, but I would like some help resolving an issue with latency. Here's my setup: 3 RHEL machines running Ignite in "server" mode, with a single partitioned cache 1 RHEL machine running Ignite in "client" mode. >From my Ignite client I use affinityCall() to co-locate with a specific cache key on one of the 3 servers. I first insert 100,000 unique keys into the cache using affinityCall(). Then, I asynchronously affinityCall() to do 10 get/modify/put on each key in parallel (with transactions), for all 100,000 keys. For the transactional workload, I see ok throughput (around 22k transactions/sec). However, my average latency for the affinityCall() is unbearably high (at 67.9 ms). I am measuring this latency in my Ignite client from right before affinityCall() to when the future's listener runs. In the server nodes, I see the average time taken by my call() function to be around 1.5 ms. That means that my own code that is executing on the server within call() is not the issue. All machines are in the same data center, and ping is around 0.25 ms between each of the 4 machines, so I don't believe network latency is the issue either. I also see the min latency is great, at around 240 microseconds. There's a big spread. What would cause such variation in latency and such a high average? Latency: Avg=67.921748 Min=0.240000 Max=1259.326000 StdDev=83.476537 Any help or suggestions would be greatly appreciated! Here is my ignite config and cache config: IgniteConfiguration [gridName=null, pubPoolSize=48, sysPoolSize=48, mgmtPoolSize=4, igfsPoolSize=24, utilityCachePoolSize=48, utilityCacheKeepAliveTime=10000, marshCachePoolSize=48, marshCacheKeepAliveTime=10000, p2pPoolSize=2, ggHome=null, ggWork=null, mbeanSrv=null, nodeId=null, marsh=null, marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, clockSyncSamples=8, clockSyncFreq=120000, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=null, evtSpi=null, colSpi=null, deploySpi=null, swapSpaceSpi=null, indexingSpi=null, addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=1, txCfg=org.apache.ignite.configuration.TransactionConfiguration@39a054a5, cacheSanityCheckEnabled=false, discoStartupDelay=60000, deployMode=PRIVATE, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@71bc1ae4, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=0], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null] CacheConfiguration [name=Orders, storeConcurrentLoadAllThreshold=5, rebalancePoolSize=2, rebalanceTimeout=10000, evictPlc=null, evictSync=false, evictKeyBufSize=1024, evictSyncConcurrencyLvl=4, evictSyncTimeout=10000, evictFilter=null, evictMaxOverflowRatio=10.0, eagerTtl=true, dfltLockTimeout=0, startSize=100000, nearCfg=null, writeSync=PRIMARY_SYNC, storeFactory=null, keepBinaryInStore=false, loadPrevVal=false, aff=null, cacheMode=PARTITIONED, atomicityMode=TRANSACTIONAL, atomicWriteOrderMode=CLOCK, backups=0, invalidate=false, tmLookupClsName=null, rebalanceMode=NONE, rebalanceOrder=0, rebalanceBatchSize=524288, rebalanceBatchesPrefetchCount=2, offHeapMaxMem=-1, swapEnabled=false, maxConcurrentAsyncOps=1000000, writeBehindEnabled=false, writeBehindFlushSize=10240, writeBehindFlushFreq=5000, writeBehindFlushThreadCnt=1, writeBehindBatchSize=512, memMode=OFFHEAP_VALUES, affMapper=null, rebalanceDelay=0, rebalanceThrottle=0, interceptor=null, longQryWarnTimeout=3000, readFromBackup=true, nodeFilter=null, sqlEscapeAll=false, sqlOnheapRowCacheSize=10240, snapshotableIdx=false, cpOnRead=true, topValidator=null] -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/High-latency-using-affinityCall-from-client-tp2931.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
