ignite persistence size growing

[kvenkatramtreddy]

Hi Team,

I have enabled persistence with cache eviction of 8 days of creation time.
But my persistence data folder is keep on growing.

Even we are deleting entries from Caches, but persistence size still
growing.

Is there way we can clean and reduce the size of persistence.

Thanks & Regards,
Venkat






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

CacheJdbcBlobStore loadCache (Hot loading data into cache)

[kvenkatramtreddy]

Hi Team,

I am using Apache Ignite 2.7
It seems to be hot loading is not available for CacheJdbcBlobStore using
loadCache. I have tried multiple times and but nothing is working.

When I looked into implementation it is extending CacheStoreAdapter and it
got empty implementation.

So please let me know how to do a hot loading using CacheJdbcBlobStore.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to map keys for cache (all partition nodes left the grid)

[kvenkatramtreddy]

Hi,

I want to receive the data even on one node. Can I add any set of commands
so that it can copy all partitions to active nodes when node left

(OR)

Can I make it is cache mode replicated and will it work?

Thanks & Regards,
Venkat





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi Evgenii,

Thank you very much for your help. it is reduced to 5 mins now. I believe,
we should have some documentation on partition count as per the data size.

Currently Ignite is showing my native persistence size around 800 MB, so can
I still decrease the partition count to improve the startup of the ignite
cluster.

following is metric

^-- Node [id=5624a0e4, uptime=2 days, 00:40:03.756]
^-- H/N/C [hosts=3, nodes=3, CPUs=24]
^-- CPU [cur=6.03%, avg=5.48%, GC=0.03%]
^-- PageMemory [pages=187791]
^-- Heap [used=533MB, free=82.63%, comm=3072MB]
^-- Off-heap [used=742MB, free=43.95%, comm=1324MB]
^--   sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^--   default region [used=741MB, free=27.57%, comm=1024MB]
^--   metastoreMemPlc region [used=0MB, free=99.56%, comm=100MB]
^--   TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=777MB]
^--   sysMemPlc region [used=0MB]
^--   default region [used=777MB]
^--   metastoreMemPlc region [used=unknown]
^--   TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]


2) We are receiving an exception "Failed to map keys for cache (all
partition nodes left the grid) " when 2 nodes left and one is running. So
please could you help me with this error as well.
I have created new topic for it.

http://apache-ignite-users.70518.x6.nabble.com/Failed-to-map-keys-for-cache-all-partition-nodes-left-the-grid-tt25964.html

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Failed to map keys for cache (all partition nodes left the grid)

[kvenkatramtreddy]

Hi Team,

I have 3 node server mode cluster setup with Cache mode Partitioned with 1
backup. We are receiving above error as soon as 2 nodes left from cluster.

All 3 nodes are on baselinetopology.

Collection baselineNodes =
ignite.cluster().currentBaselineTopology();
if(!CollectionUtils.isEmpty(baselineNodes) && baselineNodes.size() <
3){
// Get all server nodes that are already up and running.
Collection nodes =
ignite.cluster().forServers().nodes();

// Set the baseline topology that is represented by these nodes.
ignite.cluster().setBaselineTopology(nodes);

}


Exception:

org.apache.ignite.cache.CacheServerNotFoundException: Failed to map keys for
cache (all partition nodes left the grid) [topVer=AffinityTopologyVersion
[topVer=13, minorTopVer=0], cache=users]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1321)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1758)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
at
com.ibm.mobilefirst.cachemanager.LufthansaCacheDelegateImpl.getSubscribedFlights(LufthansaCacheDelegateImpl.java:477)
at
com.ibm.mobilefirst.das.notification.impl.NotificationSubscriberServiceImpl.sendMessageUpdate(NotificationSubscriberServiceImpl.java:120)
at
com.ibm.mobilefirst.das.notification.impl.NotificationSubscriberServiceImpl.processAndSend(NotificationSubscriberServiceImpl.java:65)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at
com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:95)
at
com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:154)
at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:80)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:811)
Caused by: class
org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException:
Failed to map keys for cache (all partition nodes left the grid)
[topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], cache=users]

Please help us. We required to run single node as well when it is required.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi,

Sorry to bother you. All Data in Ignite native persistence, so please could
you guide me how to reload the data again.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

hi Team,

any update or clue on above issue.

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi ezhuravlev,

I have waited for more than 2 hours and captured the logs. please find
attached log in the post.

nodeStartup.log
 
.

Please could you provide me the guidance to fix it.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi,

This time it is got hung. so I have killed the nodes and restarted again.
Please find attached logs on startup.

node1.log
  
node2.log
  

As you mentioned in your earlier reply, it seems to be an issue WAL
applyLastUpdate changes. it is taking time there. 

Is there anyway, can we write into disk as much as possible while cluster
Online.

fyi: second time restart I was unable to catch the logs, I have started the
nodes and left it and logs are overridden.

I was able to find the following log statement 

[11/29/18 5:48:23:255 UTC] 0090
.processors.cache.persistence.GridCacheDatabaseSharedManager I Finished
applying WAL changes [updatesApplied=0, time=1984500ms]


Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi,
Please find attached log.
ignite_deactivate.log

  

whole cluster is de-activated and it is going in hung state.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi,

I have deActivated the cluster and closed, but still same issue persists.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Heap size is 3 GB.
And MaxDirectMemorySize is 1.5 GB.

My cache size is around 2 GB and on Disk around 5-6 GB maximum. My total
cache elements are around 50-60 thousands.

I will receive around 700-800 thousand messages for a day and I will update
that in cache.

So please could you let me know recommended Heap and Offheap settings for my
requirement so that it can start and faster.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Please help.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startup is very slow

[kvenkatramtreddy]

Hi Team,

any update on above ticket. any hints to investigate further. As per me, it
seems to be Partition Map exchange. But I am unable to understand why it is
taking time on startup. It is got only 3 nodes.

I had the ran the SAR command as well, it was showing iowait as 6-10%.



Even it is taking time on first node start up as well.

Can we do anything like, background thread to re-balance Partition map
exchange when cluster is online.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startup is very slow

[kvenkatramtreddy]

Hi Team,

Please could any body help me or guide me to solve this issues.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

No, As you can see in the configuration it is ATOMIC mode. I did not write
any transactional code, just get and put and scanQuery.

Native persistence is enabled.

Most of the writes done by one node and other 2 nodes mostly read and less
write. 

It is got only 3 nodes. 

each node is 4 CPU and 8 GB RAM





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite startu is very slow

[kvenkatramtreddy]

Hi Team,

I am using Snapshot version of 2.7(Ignite ver.
2.7.0.20180806#19700101-sha1:DEV). Startup time is having between 8-30 mins.

My complete data set also very small, it is around 3 GB to 4 GB on Disk. 



Please find attached logs and configuration in it.

example-ignite.xml
  
ignite.log
  

FYI: I am using this snapshot because 2.6 was giving page corrupted or
Upperboud errors. It is stopped as soon as I have updated it to 2.7
Developer snapshot build.

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: ScanQuery failing on Client Mode

[kvenkatramtreddy]

Hi,

it is available. I can call other methods in the same service. Cache.query
is working in server nodes, but not on client nodes.

But ARCacheDelegateImpl is not available in server node. will it causing the
issue?.

One more questions, Can we run the few server nodes with persistence and few
of them without persistence in cluster?

Thanks & Regards,
Venkat







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

ScanQuery failing on Client Mode

[kvenkatramtreddy]

Hi Team,

ScanQuery is failing on ClientMode. below is the error. ARCacheDelegateImpl
is my service class and it is available in my classpath as well.

[err] javax.cache.CacheException: class
org.apache.ignite.IgniteCheckedException: Query execution failed:
GridCacheQueryBean [qry=GridCacheQueryAdapter [type=SCAN, clsName=null,
clause=null,
filter=com.ibm.mobilefirst.travel.VRCacheDelegateImpl$$Lambda$37/7E4A8AE0@e21cc837,
transform=com.ibm.mobilefirst.travel.VRCacheDelegateImpl$$Lambda$38/BA044000@b30f08e4,
part=null, incMeta=false, metrics=GridCacheQueryMetricsAdapter
[minTime=9223372036854775807, maxTime=0, sumTime=0, avgTime=0.0, execs=0,
completed=0, fails=0], pageSize=1024, timeout=0, incBackups=false,
dedup=false,
prj=org.apache.ignite.internal.cluster.ClusterGroupAdapter@585f62de,
keepBinary=false, subjId=f8fea35b-1944-4612-8237-b7d9f492d6dc, taskHash=0],
rdc=null,
trans=com.ibm.mobilefirst.travel.VRCacheDelegateImpl$$Lambda$38/BA044000@b30f08e4]
[err]   at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1303)
[err]   at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryFutureAdapter.next(GridCacheQueryFutureAdapter.java:168)
[err]   at
org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$5.onHasNext(GridCacheDistributedQueryManager.java:633)
[err]   at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
[err]   at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
[err]   at
org.apache.ignite.internal.processors.cache.QueryCursorImpl.getAll(QueryCursorImpl.java:114)
[err]   at
com.ibm.mobilefirst.travel.VRCacheDelegateImpl.getAssignedUserKey(VRCacheDelegateImpl.java:67)
[err]   at
com.ibm.mobilefirst.VrCacheResource.unprotected(VrCacheResource.java:154)
[err]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[err]   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
[err]   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
[err]   at java.lang.reflect.Method.invoke(Method.java:508)
[err]   at
org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:181)
[err]   at [internal classes]
[err]   at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
[err]   at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268)
[err]   at
com.ibm.mfp.server.java.adapter.shared.JAXRSSandbox$3.doFilter(JAXRSSandbox.java:579)
[err]   at
com.ibm.mfp.server.java.adapter.shared.FilterChainImpl.doFilter(FilterChainImpl.java:86)
[err]   at
com.ibm.mfp.server.java.adapter.shared.JAXRSSandbox.handleRequest(JAXRSSandbox.java:584)
[err]   at
com.ibm.mfp.server.java.adapter.internal.rest.AdaptersEndpoint.adapterServing(AdaptersEndpoint.java:115)
[err]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[err]   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
[err]   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
[err]   at java.lang.reflect.Method.invoke(Method.java:508)
[err]   at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
[err]   at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
[err]   at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
[err]   at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
[err]   at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
[err]   at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
[err]   at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
[err]   at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
[err]   at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
[err]   at
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
[err]   at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
[err]   at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
[err]   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
[err]   at
com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1287)
[err]   at [internal classes]
[err]   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
[err]   at

Re: Too Many open files (native persistence bin files)

[kvenkatramtreddy]

one correction: lsof showing the information for threads as well, so that is
why it is showing number of Threads * file descriptor.

My current ulimit is 32635, open files in 6000 but still we are receiving
too many files open.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Too Many open files (native persistence bin files)

[kvenkatramtreddy]

Hi Team,

We have got one production issue using Ignite. as per the lsof (File
descriptors ) it is around a 1 million open files with given process and all
used by ignite.

I am running ignite as emebedding into my web application with native
persistence enabled with 3 nodes as Cache Replicated mode.

I am using ignite-direct-io as well.

We are running our web app in Websphere Liberty Profile.

when I run below command in lsof -p  | grep
cache-turnAroundProcesses/part-31.bin | wc -l

result is 189. One bin file is opened around 189 times.


openfiles_bin.txt
  

Thanks & Regards,
Venkat









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Failed to add node to topology because it has the same hash code for partitioned affinity as one of existing nodes

[kvenkatramtreddy]

Hi It is having unique consistentId across cluster. All nodes running for
some time, it is happening after some time.Please see the discoverySpi and
consistenId details below.
*Node1 ConsistentId (Server 1):*



 
127.0.0.1:48500..48520   
192.168.0.4:48500..48520   
192.168.0.5:48500..48520

Node2 ConsistentId (Server 1):



 
127.0.0.1:48500..48520   
192.168.0.4:48500..48520   
192.168.0.5:48500..48520

Node 3(server2):



 
127.0.0.1:48500..48520   
192.168.0.4:48500..48520   
192.168.0.5:48500..48520




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Failed to add node to topology because it has the same hash code for partitioned affinity as one of existing nodes

[kvenkatramtreddy]

Hi Team,

I have configured my all caches replicated mode with native persistence
enable and running it on 3 nodes. 2 nodes runs on same server and another
node run on different server.

I have configured unique consistentId for each node and unique IGNITE_HOME.
I also configured 






Receiving the Failed to add node to topology error after some time. Please
could you help us to resolve this issue.

igniteSamehashError.txt

  

Thanks & Regards,
Venkat








--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: SYSTEM_WORKER_TERMINATION (Item Not found)

[kvenkatramtreddy]

Hi, 

It seems be all of them resolved. Can I go head and download nightly build
and use it.

Or will it have any impact.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

SYSTEM_WORKER_TERMINATION (Item Not found)

[kvenkatramtreddy]

Hi,

We are still receiving Runtime Failure bound (Item not found) in 2.6 as
well. We are receiving this error everyday midnight check point.

We have around 15 caches and configure expiry as 3 days, 8 days and 2 caches
are eternal.



Please find attached checkpoint log and error in below links. I have
attached configuration as well.

http://apache-ignite-users.70518.x6.nabble.com/file/t1700/example-ignite.xml

apacheIgniteError.txt

  

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Runtime failure on bounds

[kvenkatramtreddy]

Hi, 

We have received Runtime failure on bounds again in 2 days. I am using
ignite 2.6 and started all nodes from scratch ( removed existing data).

error below

[7/21/18 13:22:43:031 UTC] 008f 

E Critical system error detected. Will be handled accordingly to configured
handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow [
class org.apache.ignite.IgniteException: Runtime failure on bounds:
[lower=null, upper=PendingRow []]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:785)
Caused by: java.lang.IllegalArgumentException: Invalid object type: 0
at
org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:166)
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toKeyCacheObject(CacheObjectBinaryProcessorImpl.java:848)
at
org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toKeyCacheObject(IgniteCacheObjectProcessorImpl.java:219)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteKey(CacheDataRowAdapter.java:379)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:229)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:174)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4660)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4562)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4501)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:927)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:959)
... 6 more

[7/21/18 13:22:43:046 UTC] 008f 

E JVM will be halted immediately due to the failure:
[failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow [

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Runtime failure on bounds

[kvenkatramtreddy]

Hi Andrei,

Yes, I have stopped all the servers and removed corrupted nodes data and
upgraded to 2.6 and restart the server.

any more suggestion and configuration changes to prevent this issue.



Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Runtime failure on bounds

[kvenkatramtreddy]

Hi Team,

We are receiving Runtime failure on bounds even after updating it to 2.6
version. 

We used to receive Failed to get page IO instance (page content is
corrupted) from 2.3, 2.4 and 2.5, now in 2.6 it is changed to Item not
found: 1.

Till now we are in test environments but we are planning to move Production
now. Please could you help us a work around or fix for this issue.

Please find attached gc-log, excetion, and configuration


example-ignite.xml
  

gc-log.txt
  
messages_copy.log
  

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Node stopped automatically

[kvenkatramtreddy]

Please could you provide some pointers, so that I can look deeper into the
issue. 

1) Does PageMemory cause any issues

as per the metrics, one node is contains only 70  and other node is
PageMemory [pages=390178]

2) Why whole cluster is going down, when one node is down

3) Can I restart the ignite programatically if this error occurs.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Node stopped automatically

[kvenkatramtreddy]

Ignite related only that log,  
After that only  cache stopped exception in the log. 

Even we have enabled -DIGNITE_QUIET=false, but it is printing that much
only.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Node stopped automatically

[kvenkatramtreddy]

Hi,

Node stopped automatically after 48 hours uptime. Please find the logs
below.

I got 2 hosts and 3 Ignite nodes 


Host 1, Node 1



Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=5582ed60, uptime=48:00:04.981]
^-- H/N/C [hosts=2, nodes=3, CPUs=4]
^-- CPU [cur=15.67%, avg=15.01%, GC=0.43%]
^-- PageMemory [pages=390178]
^-- Heap [used=1387MB, free=11.75%, comm=1572MB]
^-- Non heap [used=369MB, free=-1%, comm=409MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]



[6/27/18 8:28:43:098 UTC] 00c5
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  I TCP discovery
accepted incoming connection [rmtAddr=/57.20.44.84, rmtPort=47056]
[6/27/18 8:28:43:098 UTC] 00c5
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  I TCP discovery
spawning a new thread for connection [rmtAddr=/57.20.44.84, rmtPort=4
7056]

[6/27/18 8:28:43:117 UTC] 000d83ee
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  I Started
serving remote node connection [rmtAddr=/57.20.44.84:47056, rmtPort=47056]
[6/27/18 8:28:43:128 UTC] 00c4
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  W Node is out
of topology (probably, due to short-time network problems).
[6/27/18 8:28:43:129 UTC] 00cb
ache.ignite.internal.managers.discovery.GridDiscoveryManager W Local node
SEGMENTED: TcpDiscoveryNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, ad
drs=[57.20.44.85], sockAddrs=[myhost1.org/57.20.44.85:48500],
discPort=48500, order=2, intOrder=2, lastExchangeTime=1530088123126,
loc=true, ver=2.4.0#201803
05-sha1:aa342270, isClient=false]
[6/27/18 8:28:43:135 UTC] 00c5
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  I TCP discovery
accepted incoming connection [rmtAddr=/57.20.44.84, rmtPort=58172]
[6/27/18 8:28:43:137 UTC] 000d83ee
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  I Finished
serving remote node connection [rmtAddr=/57.20.44.84:47056, rmtPort=47056
[6/27/18 8:28:43:140 UTC] 00c5
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  I TCP discovery
spawning a new thread for connection [rmtAddr=/57.20.44.84, rmtPort=5
8172]
[6/27/18 8:28:43:140 UTC] 000d83f0
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  I Started
serving remote node connection [rmtAddr=/57.20.44.84:58172, rmtPort=58172]
[6/27/18 8:28:43:162 UTC] 000d83f0
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  I Finished
serving remote node connection [rmtAddr=/57.20.44.84:58172, rmtPort=58172
[6/27/18 8:28:43:184 UTC] 00cb
ache.ignite.internal.managers.discovery.GridDiscoveryManager W Stopping
local node according to configured segmentation policy.




Host 2(have 2 ignite nodes) Node1

Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=58a3d911, uptime=48:00:05.572]
^-- H/N/C [hosts=2, nodes=3, CPUs=4]
^-- CPU [cur=8.17%, avg=8.27%, GC=0.5%]
^-- PageMemory [pages=70]
^-- Heap [used=1306MB, free=16.92%, comm=1572MB]
^-- Non heap [used=355MB, free=-1%, comm=392MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]



[6/27/18 8:28:34:381 UTC] 0095
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  W Timed out
waiting for message delivery receipt (most probably, the reason is in lon
g GC pauses on remote node; consider tuning GC and increasing 'ackTimeout'
configuration property). Will retry to send message with increased timeout
[currentTimeout=1, rmtAddr=
myhost2.org/57.20.44.85:48500, rmtPort=48500]
[6/27/18 8:28:34:382 UTC] 0095
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  W Failed to
send message to next node [msg=TcpDiscoveryMetricsUpdateMessage [super=Tc
pDiscoveryAbstractMessage [sndNodeId=null,
id=27b0fa80461-58a3d911-aa4c-453d-ae5a-0fc3d450a09f,
verifierNodeId=58a3d911-aa4c-453d-ae5a-0fc3d450a09f, topVer=0, pendingIdx=0,
failedNo
des=null, isClient=false]], next=TcpDiscoveryNode
[id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, addrs=[57.20.44.85],
sockAddrs=[myhost2.org/57.20.44.85:48500], d
iscPort=48500, order=2, intOrder=2, lastExchangeTime=1529051134737,
loc=false, ver=2.4.0#20180305-sha1:aa342270, isClient=false], errMsg=Failed
to send message to next node [msg=Tcp
DiscoveryMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=27b0fa80461-58a3d911-aa4c-453d-ae5a-0fc3d450a09f,
verifierNodeId=58a3d911-aa4c-453d-ae5a-0fc3d45
0a09f, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
next=ClusterNode [id=5582ed60-76f5-42b1-bd19-ed3af12b12f7, order=2,
addr=[57.20.44.85], daemon=false]]]
[6/27/18 8:28:34:480 UTC] 0095
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi  W Local node
has detected failed nodes and started cluster-wide procedure. To speed u
p failure 

Re: ClusterTopologyServerNotFoundException

[kvenkatramtreddy]

Hi,

Thank you very much.

We are receiving this error only when one node was running. Exception is
gone as soon other nodes are started and joined the cluster.

I have attached the single node where we received the error.

exception.log
  

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache is too slow

[kvenkatramtreddy]

Hi,

Thank you very much for your reply. I found that it is an issue with
Webserver. I can see now Ignite cache is giving results are around
1-millisecond to 14 milliseconds with onheapEnabled and without
onheapEnabled, results are around 3 milliseconds to 200 milliseconds. 

Are these expected results , (or) still can I fine tune ignite cache.

I can see 0 milliseconds with ConcurrentMap.

Thanks & Regards,
Venkat.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Cache is too slow

[kvenkatramtreddy]

Hi,We have setup 3 node cluster with Replicated caches. Ignite starts with
our JAX-RS servvices on Liberty server.We have exposed an API and reading
the values from cache(no business logic). Just read it from Ignite cache and
return it and it is taking more than 6 seconds. We had run load test with
Gatling with 300 users and 10 seconds ramp up.Cache data also very small, it
is around 200BAll the requests are sent to same node.Gatling results
 
System configuration2 core CPU8 GB RAMRHELConfiguration can found following
link 
http://apache-ignite-users.70518.x6.nabble.com/file/t1700/example-ignite-sample.xml

  

Please could help us to improve the performance. We are looking for read
performance.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

ClusterTopologyServerNotFoundException

[kvenkatramtreddy]

Hi,We have setup 3 node cluster on Linux server with 2 core CPU and 8 GB RAM.
We have configured Ignite persistence and replicated caches.Please find
attached configuration in the email. example-ignite-sample.xml

 
We are receiving following error when we start single node.class
org.apache.ignite.IgniteException: Failed to map keys for cache (all
partition nodes left the grid) [topVer=AffinityTopologyVersion [topVer=6,
minorTopVer=0], cache=users]at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.containsKey(GridCacheAdapter.java:633)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.containsKey(IgniteCacheProxyImpl.java:972)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.containsKey(GatewayProtectedCacheProxy.java:816)
at
com.ibm.PushNotificationService.isAssignedUser(PushNotificationService.java:239)
at com.ibm.PushNotificationService.run(PushNotificationService.java:102)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at com.ibm.ws.concurrent.internal.SubmittedTask.run(SubmittedTask.java:276)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by: class
org.apache.ignite.internal.cluster.ClusterTopologyServerNotFoundException:
Failed to map keys for cache (all partition nodes left the grid)
[topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], cache=users] at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.serverNotFoundError(GridPartitionedSingleGetFuture.java:711)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:332)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:216)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.init(GridPartitionedSingleGetFuture.java:208)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync0(GridDhtAtomicCache.java:1391)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1600(GridDhtAtomicCache.java:130)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:469)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:467)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:756)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync(GridDhtAtomicCache.java:467)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.containsKeyAsync(GridCacheAdapter.java:641)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.containsKey(GridCacheAdapter.java:630)
... 11 moreMy requirement is that, it should run on available nodes, when
other nodes available it should be replicated.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Delete Message Topic and Listener

[kvenkatramtreddy]

I have setup ignite cluster with native persistence. I enabled messaging and
created a message topic and added listener to it and deployed.

There were some business logic errors in the listener code and I fixed it.
But the changes are not yet all reflecting. I have tried restarting the
ignite server multiple times, but no use.

Please could you help us here.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/