Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi Evgenii,

Thank you very much for your help. it is reduced to 5 mins now. I believe,
we should have some documentation on partition count as per the data size.

Currently Ignite is showing my native persistence size around 800 MB, so can
I still decrease the partition count to improve the startup of the ignite
cluster.

following is metric

^-- Node [id=5624a0e4, uptime=2 days, 00:40:03.756]
^-- H/N/C [hosts=3, nodes=3, CPUs=24]
^-- CPU [cur=6.03%, avg=5.48%, GC=0.03%]
^-- PageMemory [pages=187791]
^-- Heap [used=533MB, free=82.63%, comm=3072MB]
^-- Off-heap [used=742MB, free=43.95%, comm=1324MB]
^--   sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^--   default region [used=741MB, free=27.57%, comm=1024MB]
^--   metastoreMemPlc region [used=0MB, free=99.56%, comm=100MB]
^--   TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=777MB]
^--   sysMemPlc region [used=0MB]
^--   default region [used=777MB]
^--   metastoreMemPlc region [used=unknown]
^--   TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]


2) We are receiving an exception "Failed to map keys for cache (all
partition nodes left the grid) " when 2 nodes left and one is running. So
please could you help me with this error as well.
I have created new topic for it.

http://apache-ignite-users.70518.x6.nabble.com/Failed-to-map-keys-for-cache-all-partition-nodes-left-the-grid-tt25964.html

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[Evgenii Zhuravlev]

How about to create a new cache with the new partition count and move all
the data from the previous one? After that repeat this process for each
cache.

Best Regards,
Evgenii

вт, 4 дек. 2018 г. в 21:14, kvenkatramtreddy :

> Hi,
>
> Sorry to bother you. All Data in Ignite native persistence, so please could
> you guide me how to reload the data again.
>
> Thanks & Regards,
> Venkat
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi,

Sorry to bother you. All Data in Ignite native persistence, so please could
you guide me how to reload the data again.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[Evgenii Zhuravlev]

Hi,

I'd say that 128 partitions should be more than enough for 3 nodes, just
try it.

Note: you will need to reload all data in caches, because they will have
different affinity.

Best Regards,
Evgenii

вт, 4 дек. 2018 г. в 02:36, kvenkatramtreddy :

> Hi,
>
> Please could you suggest the number of partitions for my caches, My Cache
> size currently on Disk is 700 MB and smallest 10 MB.
>
> only 5-6 caches are 700 MB rest of them are all around 100 MB.
>
> Current I configured RendezvousAffinityFunction, let me know do you have
> recommendations here.
>
> 
> class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction"
> id="rendezvousAffinityFunction" scope="prototype">
> 
> 
>
> Thanks & Regards,
> Venkat
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite startu is very slow

[Evgenii Zhuravlev]

Hi,

You have a pretty small amount of heap and 30 caches. Each cache creates
some overhead for node startup, because node should read states for each
partition of each cache(and each partition is a separate file). You can
reduce this overhead by configuring the same cache group:
https://apacheignite.readme.io/docs/cache-groups. Also, if you have only 3
nodes, it makes sense to reduce the number of partitions for each cache, by
configuring affinity function.

Evgenii

пн, 3 дек. 2018 г. в 20:39, kvenkatramtreddy :

> hi Team,
>
> any update or clue on above issue.
>
> Thanks & Regards,
> Venkat
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite startu is very slow

[kvenkatramtreddy]

hi Team,

any update or clue on above issue.

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi ezhuravlev,

I have waited for more than 2 hours and captured the logs. please find
attached log in the post.

nodeStartup.log
 
.

Please could you provide me the guidance to fix it.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi,

This time it is got hung. so I have killed the nodes and restarted again.
Please find attached logs on startup.

node1.log
  
node2.log
  

As you mentioned in your earlier reply, it seems to be an issue WAL
applyLastUpdate changes. it is taking time there. 

Is there anyway, can we write into disk as much as possible while cluster
Online.

fyi: second time restart I was unable to catch the logs, I have started the
nodes and left it and logs are overridden.

I was able to find the following log statement 

[11/29/18 5:48:23:255 UTC] 0090
.processors.cache.persistence.GridCacheDatabaseSharedManager I Finished
applying WAL changes [updatesApplied=0, time=1984500ms]


Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[Evgenii Zhuravlev]

I've asked for the logs of node startup, not the log for stopping the
cluster.

Evgenii

вт, 27 нояб. 2018 г. в 23:19, kvenkatramtreddy :

> Hi,
> Please find attached log.
> ignite_deactivate.log
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1700/ignite_deactivate.log>
>
>
> whole cluster is de-activated and it is going in hung state.
>
> Thanks & Regards,
> Venkat
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi,
Please find attached log.
ignite_deactivate.log

  

whole cluster is de-activated and it is going in hung state.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[Evgenii Zhuravlev]

Can you share logs from a startup after deactivation?

Evgenii

пн, 26 нояб. 2018 г. в 20:58, kvenkatramtreddy :

> Hi,
>
> I have deActivated the cluster and closed, but still same issue persists.
>
> Thanks & Regards,
> Venkat
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite startu is very slow

[kvenkatramtreddy]

Hi,

I have deActivated the cluster and closed, but still same issue persists.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[Evgenii Zhuravlev]

For this particular case, it's just enough to stop cluster gracefully
before starting it again. Just deactivate cluster before stopping. In this
case, when you will start the nodes again, they won't need to apply WAL
changes.

Evgenii

вс, 25 нояб. 2018 г. в 04:54, kvenkatramtreddy :

> Heap size is 3 GB.
> And MaxDirectMemorySize is 1.5 GB.
>
> My cache size is around 2 GB and on Disk around 5-6 GB maximum. My total
> cache elements are around 50-60 thousands.
>
> I will receive around 700-800 thousand messages for a day and I will update
> that in cache.
>
> So please could you let me know recommended Heap and Offheap settings for
> my
> requirement so that it can start and faster.
>
> Thanks & Regards,
> Venkat
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite startu is very slow

[kvenkatramtreddy]

Heap size is 3 GB.
And MaxDirectMemorySize is 1.5 GB.

My cache size is around 2 GB and on Disk around 5-6 GB maximum. My total
cache elements are around 50-60 thousands.

I will receive around 700-800 thousand messages for a day and I will update
that in cache.

So please could you let me know recommended Heap and Offheap settings for my
requirement so that it can start and faster.

Thanks & Regards,
Venkat



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[Evgenii Zhuravlev]

Hi,

How many heap do you have?

>From logs I see, that before this, cluster was not stopped properly and the
checkpoint was not saved to the disk. Because of that, after the start,
nodes started to applying WAL changes(
https://apacheignite.readme.io/docs/write-ahead-log). It took 1491578ms,
probably because you have a very small heap size. Also, usually messages
like "Possible too long JVM pause: 2710 milliseconds." is a symptom of a
long GC pauses.

Evgenii

пт, 23 нояб. 2018 г. в 08:04, kvenkatramtreddy :

> Please help.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite startu is very slow

[kvenkatramtreddy]

Please help.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[kvenkatramtreddy]

No, As you can see in the configuration it is ATOMIC mode. I did not write
any transactional code, just get and put and scanQuery.

Native persistence is enabled.

Most of the writes done by one node and other 2 nodes mostly read and less
write. 

It is got only 3 nodes. 

each node is 4 CPU and 8 GB RAM





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite startu is very slow

[Mikael]

Hi!

This one looks a bit fishy to me:

"Failed to wait for initial partition map exchange. Possible reasons are:..."

Do you use transactions ?

Mikael

Den 2018-11-16 kl. 07:09, skrev kvenkatramtreddy:

Hi Team,

I am using Snapshot version of 2.7(Ignite ver.
2.7.0.20180806#19700101-sha1:DEV). Startup time is having between 8-30 mins.

My complete data set also very small, it is around 3 GB to 4 GB on Disk.



Please find attached logs and configuration in it.

example-ignite.xml

ignite.log


FYI: I am using this snapshot because 2.6 was giving page corrupted or
Upperboud errors. It is stopped as soon as I have updated it to 2.7
Developer snapshot build.

Thanks & Regards,
Venkat




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/