Ignite Server nodes crash with OutOfMemoryError on cluster upscale
Hello, dear Ignite users & developers I'm running an Ignite 2.6.0 cluster in k8s as a partitioned cache (off-heap) with eviction policy. It all works great and handles around 100K RPM in peaks (many thanks for the software!). The problem I'm facing is that on cluster topology change, all existing Ignite server instances crash with Java OutOfMemoryError and lose data (gladly it's cache, but still important for performance of our systems). First time I've noticed that happened, I've increased Java heap for Ignite server nodes to 3 GB, and it seemed to work for a while during minor topology changes (ie add one node, remove one node) But I just increased number of nodes to 12 from 8, so 4 new nodes, and all already running nodes crashed with OOM again Looks like the more nodes we're trying to add to a cluster at around the same time, the more memory is required for existing nodes to handle this change. Do you have any recommendations on how much Java heap an Ignite server need for a given cluster size? Note that actual data is stored off-heap (see configs below). Thanks! Configs: k8s Ignite server containers: containers: - name: ignite-node image: apacheignite/ignite:2.6.0 resources: requests: memory: "28G" cpu: 1 limits: memory: "29G" cpu: 2 JVM_OPTS: -server \ -Djava.net.preferIPv4Stack=true \ -XX:+UnlockExperimentalVMOptions \ -XX:+UseCGroupMemoryLimitForHeap \ -XX:MaxDirectMemorySize=25g \ -Xms1g \ -Xmx3g \ -XX:+UseG1GC \ -XX:+AlwaysPreTouch \ -XX:+ScavengeBeforeFullGC \ -XX:+DisableExplicitGC XML: Kind regards, Artem Zinnatullin.
Need help for setting offheap memory
Hi, We are using ignite to share in memory data across the spark jobs. . I am using below configuration to set ignite offheap memory. I would like to set it as 100 gb. However when I print the node statistics using visor it shows offheap max memory as 1 gb. Please suggest. Apache Ignite version 2.3 Thanks, Amol
Re: rest api rmdbs integration
The error means that there is no such class on classpath. Please check if that's the case or not. You can also provide your project and steps to reproduce the issue, I will take a look then. -Val -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Ignite.NET + Apache Cassandra integration for SQL cache
Hi,
We create a cache using SQL DDL syntax.
How can I configure Cassandra persistent store to the cache in .NET
application ?
It seems like default CacheConfiguration is automatically applied
internally in Java codes for SQL cache, so not able to load up Spring.XML
for that cache.
Here is the steps creating a SQL cache.
/Ignite: 2.6
OS: Windows Server 2016
Jdk: Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
Programming language: C#/
*a)*
// Create dummy cache to act as an entry point for SQL queries (new SQL API
which do not require this
// will appear in future versions, JDBC and ODBC drivers do not require it
already).
var cacheCfg = new CacheConfiguration(m_odbcCacheName)
{
SqlSchema = "PUBLIC",
CacheMode = CacheMode.Replicated
};
m_odbcCache = m_ignite.GetOrCreateCache(cacheCfg);
*b)*
// Create a VALUE_TYPE class
public class UserData : IBinarizable
{
[QuerySqlField]
public DateTime LastUpdated { get; set; }
[QuerySqlField]
public byte[] Data { get; set; }
public UserData() { }
public UserData(byte[] btUserData)
{
LastUpdated = DateTime.UtcNow;
Data = btUserData;
}
public void WriteBinary(IBinaryWriter writer)
{
writer.WriteTimestamp(nameof(LastUpdated), LastUpdated);
writer.WriteByteArray(nameof(Data), Data);
}
public void ReadBinary(IBinaryReader reader)
{
LastUpdated = reader.ReadTimestamp(nameof(LastUpdated)) ??
DateTime.UtcNow;
Data = reader.ReadByteArray(nameof(Data));
}
}
*c)*
// Crete an actual UserData cache by running a DDL query on entry point
cache.
var qry = @"
CREATE TABLE IF NOT EXISTS [name]
(
CacheKey VARCHAR,
AffinityKey LONG,
LastUpdated TIMESTAMP,
Data VARBINARY(MAX),
PRIMARY KEY (CacheKey, AffinityKey)
)
WITH ""BACKUPS=2, TEMPLATE=PARTITIONED,
WRITE_SYNCHRONIZATION_MODE=PRIMARY_SYNC, AFFINITYKEY=AffinityKey,
VALUE_TYPE=UserData""
";
var qryResults = m_odbcCache.Query(new SqlFieldsQuery(qry)).GetAll();
Thanks
Sam
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
rest api rmdbs integration
i have integrated a database with ignite and the generated project created
classes for the tables. If i try run Sql scan query execute over the cache
like this example
http://host:port/ignite?cmd=qryscanexe=10=Person=org.apache.ignite.schema.Person
here is my query string
http://127.0.0.1:8080/ignite?cmd=qryscanexe=10=ComplianceSummaryCache=com.demo.mydbdemo.compliancesummary
i get
{"successStatus":1,"error":"Failed to find target class:
com.demo.mydbdemo.compliancesummary","response":null,"sessionToken":null}
the project was generated by the web console and these classes are in there
so they should work but it can't find it so any ideas what is going on here?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Cluster with remote server + Cassandra Persistence
Hi, I am basically starting Ignite node through my Java project. When I run the above specified code, I first run the ignite.sh from my remote server as well. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite backups on collocated data
Hello, Yes, backup copies will also be collocated. Thanks, S. пт, 27 июл. 2018 г. в 15:11, kotamrajuyashasvi : > Hi > > Suppose I have two ignite caches and these two caches are collocated based > on a field using AffinityKemapped. Now if I configure same number of > backups > for both the caches, do the backup copies also get collocated i.e backed up > on same nodes ? > > For ex: one row of cache1 and one row from cache2 are collocated i.e will > be > in same node/partition. Now will the backup copies of these rows also > guaranteed to be present on same nodes/partitions ? Or can be on different > nodes/partitions. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Ignite backups on collocated data
Hi Suppose I have two ignite caches and these two caches are collocated based on a field using AffinityKemapped. Now if I configure same number of backups for both the caches, do the backup copies also get collocated i.e backed up on same nodes ? For ex: one row of cache1 and one row from cache2 are collocated i.e will be in same node/partition. Now will the backup copies of these rows also guaranteed to be present on same nodes/partitions ? Or can be on different nodes/partitions. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: IgniteMessaging, What am I doing wrong?
No, a new client will not receive these messages because they were sent on
previous topology version. (topology version changing after join/left any
node)
The messages will be sent only to current client nodes.
On Fri, Jul 27, 2018 at 2:07 PM monstereo wrote:
> thanks,
> When I want to send messages from server to client, it works
> probably, there were a problem as you stated.
>
> May I ask :
> Let's say server send a messages to client ("hello")
> When I create another client after seconds, minutes vs.. , this new client
> will see this messages?
> Same question for events?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: The problem after the ignite upgrade
I opened the persistent storage cluster, version 2.3 as long as the boot server and then activate the cluster when the client can be connected directly, after upgrading to 2.6, after activation server cluster for the first time the client connection complains, you then need to Deactive the cluster with clients to connect again later, in an error once, then again to reactivate can connect successful, why? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: IgniteMessaging, What am I doing wrong?
thanks,
When I want to send messages from server to client, it works
probably, there were a problem as you stated.
May I ask :
Let's say server send a messages to client ("hello")
When I create another client after seconds, minutes vs.. , this new client
will see this messages?
Same question for events?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The problem after the ignite upgrade
Hi, Could you please provide step by step instruction on how you upgrade the cluster? Please note that in case if you try to use the persistence then the important feature that was added in 8.4 - baseline topology. A baseline topology is a set of Ignite server nodes intended to store data. The nodes from the baseline topology are not limited in terms of functionality and behave as regular server nodes that act as a container for data and computations in Ignite. It changes the activation process. You can read about it here: https://apacheignite.readme.io/docs/baseline-topology BR, Andrei -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Delete/remove cache does not free the memory in PCF server
Hi,
Do you use the in-memory region for these caches or region with persistence?
What is your ignite version?
On Fri, Jul 27, 2018 at 12:17 AM okiesong wrote:
> Hi, this is basically the command that I am using to clear and destroy the
> cache from the server that is deployed on the PCF server.
>
> But memory does not free, even after I do this.
>
> private clearCacheByName(String item) {
> if (ignite.cacheNames().contains(item)) {
> ignite.cache(item).clear();
> ignite.cache(item).destroy();
> ignite.destroyCache(item);
> }
> }
>
> When I call ignite.cacheNames(), that specified cache is gone, but memory
> is
> not freed from the PCF server that runs Ignite.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: IgniteMessaging, What am I doing wrong?
Hi,
Are you sure that server sends a message only after a client is joined to
topology?
Please, check that ClusterGroup clientNodes is not empty before sending a
message to the topic.
On Fri, Jul 27, 2018 at 8:27 AM monstereo wrote:
> I want to send message to client nodes from server nodes, and client nodes
> will take this message and print it.
>
> Here is the server nodes
>
> public static void main {
> Ignite server = //start ignite with default config path
> ClusterGroup clientNodes = server.cluster().forClients();
> IgniteMessaging messaging = server.message(clientNodes);
> messaging.send("ServerTopic", "hello");
> }
>
> Here is the client node
> public static void main ...{
> Ignite client = //start ignite client with default config
> ClusterGroup clientNodes = client.cluster().forClients();
> IgniteMessaging serverTopic = client.message(clientNodes);
> consumerTopic.remoteListen("ServerTopic", (nodeId, message) ->{
> System.out.println("Severr with id:" + nodeId +"\nsend this
> message : " + message);
> return true;
> });
> }
>
> But I can not see the message on client nodes.
> (I run client first, then run the server node(s))
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Client Server peer class loading
Here is the link to the documentation which contains parameter that you can use: https://apacheignite.readme.io/docs/yarn-deployment#section-configuration 2018-07-27 12:42 GMT+03:00 debashissinha : > Hi ezhuravlev , > > Can you also please tell me a way to package my configuration classes as a > yarn jar > > > Thanks & Regards > Debashis Sinha > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Client Server peer class loading
Hi ezhuravlev , Can you also please tell me a way to package my configuration classes as a yarn jar Thanks & Regards Debashis Sinha -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Client Server peer class loading
Thanks a lot for your advice -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Client Server peer class loading
Hi, PeerClassLoading is not working for classes which are part of a cache configuration. These classes (cache store implementations, eviction and expiry policies, etc.) must be explicitly deployed on all server nodes. Evgenii 2018-07-27 11:05 GMT+03:00 debashissinha : > Hi , > I have a situation here . Whereby there is a cachestore adapter class for > cache read through in my code and I am starting the ignite cluster in > client mode. > > I am connecting to ignite cache server on yarn . I have specified the > server > to load peerclassloading and the same is enabled in client application also > . > > However when I call the getcache method on client node it gives a class not > found error on server node. > > I was of the opinion that if peerclassloading is enabled in both client and > server node then any class in client node will be loaded when server node > is > called .. > > Can you suggest where I am going wrong. > > Thanks in advance. > Debashis Sinha > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Client Server peer class loading
Hi , I have a situation here . Whereby there is a cachestore adapter class for cache read through in my code and I am starting the ignite cluster in client mode. I am connecting to ignite cache server on yarn . I have specified the server to load peerclassloading and the same is enabled in client application also . However when I call the getcache method on client node it gives a class not found error on server node. I was of the opinion that if peerclassloading is enabled in both client and server node then any class in client node will be loaded when server node is called .. Can you suggest where I am going wrong. Thanks in advance. Debashis Sinha -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
The problem after the ignite upgrade
The problem with the Ignite version upgrading from 2.3 to 2.6 is that when the Server side starts up, the client will report an error when connecting to the Server side. The client must first stop activating the Server side, then connect with the client once, and then reactivate the Server side before the client can connect properly. Why? Is this a BUG -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
