Ignite pod keeps crashing and failed to recover the node
Ignite has been deployed on the kubernets , there are 3 replicas of server
pod. The pods were up and running fine for 9 days. We have created 180
inventory tables and 204 transactional tables. The data has been
inserted using the PyIgnite client using the cache.put() method. This is a
very slow operation because PyIgnite is very slow. Each insert is
committed one at a time, so it is not able to do bulk-style inserts. The
PyIgnite was inserting about 20 of the inventory tables simultaneously (20
different threads/processes).
The cluster was nowhere stable after 9days, one of the pod crashed and
failed to recover. Below is the error log:
{"type":"log","host":"ignite-cluster-ignite-esoc-2","level":"ERROR","system":"ignite-service","time":"2019-08-16T17:13:34,769Z","logger":"GridCachePartitionExchangeManager","timezone":"UTC","log":"Failed
to process custom exchange task: ClientCacheChangeDummyDiscoveryMessage
[reqId=6b5f6c50-a8c9-4b04-a461-49bfd0112eb0, cachesToClose=null,
startCaches=[BgwService]] java.lang.NullPointerException| at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.processClientCachesChanges(CacheAffinitySharedManager.java:635)|
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.processCustomExchangeTask(GridCacheProcessor.java:391)|
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.processCustomTask(GridCachePartitionExchangeManager.java:2475)|
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2620)|
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)|
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)|
at java.lang.Thread.run(Thread.java:748)"}
{"type":"log","host":"ignite-cluster-ignite-esoc-2","level":"WARN","system":"ignite-service","time":"2019-08-16T17:13:36,724Z","logger":"GridCacheDatabaseSharedManager","timezone":"UTC","log":"Ignite
node stopped in the middle of checkpoint. Will restore memory state and
finish checkpoint on node start."}
The error report file and ignite-config.xml has been attached for your info.
Heap Memory and RAM Configurations are as below on each of the ignite
server container:
Heap Memory: 32gb
RAM: 64GB
Default memory region:
cpu: 4
Persistence volume
wal_storage_size: 10GB
persistence_storage_size: 10GB
Thanks
With Regards
Radha
ignite-config.xml:
http://www.springframework.org/schema/beans;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xmlns:util="http://www.springframework.org/schema/util;
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd;>
Trans
Info
Msg
Unable to insert data into table after altering the table
Hi, Insertion is failing, after altering the table. Followed below steps: -> CREATE TABLE student (id_ VARCHAR, firstName VARCHAR, lastName VARCHAR, birthDate TIMESTAMP, height DOUBLE, PRIMARY KEY(id)) -> INSERT INTO student (id, firstName, lastName, birthDate, height) VALUES (1000, 'huxt', 'abx', CURRENT_TIMESTAMP(), 5.6); -> ALTER TABLE student DROP COLUMN birthDate; -> ALTER TABLE student ADD COLUMN birthDate VARCHAR; -> INSERT INTO student (id, firstName, lastName, birthDate, height) VALUES (1001, 'luan', 'qwst', '11223', 6.6); Getting below error: Error: class org.apache.ignite.binary.BinaryObjectException: Wrong value has been set [typeName=SQL_PUBLIC_STUDENT_e910ba3a_3a65_4d53_9265_8c12a9b0bac3, fieldName=BIRTHDATE, fieldType=Timestamp, assignedValueType=String] (state=5,code=1) Issues faced: -> unable to insert the data after modifying the datatype of the field in a table. -> If i do " select * from student;" , I am seeing the old data being inserted for the birthdate column, after dropping birthdate colmun and adding the colmn with the same name but different datatype. Is this a know issue? Or soemthing i am doing wrong. If it is a know issue please provide the workaround. Thanks radha
Unable to get the security context
Hi,
I have implemented the grid security processor and setting the
securityconext holder in the authenticate function as below,
public class MySecurityProcessor extends GridProcessorAdapter implements
DiscoverySpiNodeAuthenticator, GridSecurityProcessor, IgnitePlugin
{
public SecurityContext authenticate(AuthenticationContext
authenticationContext) throws IgniteCheckedException
{
SecuritySubject secureSecuritySubject = new SecuritySubject(
authenticationContext.subjectId(),
authenticationContext.subjectType(),
authenticationContext.credentials().getLogin(),
authenticationContext.address()
);
SecurityContext securityContext = new
MySecurityContext(secureSecuritySubject, accessToken);
SecurityContextHolder.set(securityContext);
return securityContext;
}
public void authorize(String name, SecurityPermission perm, SecurityContext
securityCtx) throws SecurityException {
System.out.println( SecurityContextHolder.get());
System.out.println( securityCtx );
//do some authorization
.
}
public boolean isGlobalNodeAuthentication() {
// TODO Auto-generated method stub
return false;
}
..
}
In plugin provider i am creating the component : GridSecurityProcessor.
During Rest api call:
-> when rest call is made authorise function in the security processor is
getting called twice one by the GridRestProcessor and another
GridCacheProcessor, is it mandatory to call that twice? When authorise
function is called by the GridRestProcessor security context is available
but when the GridCacheProcessor is called security context is coming as
null always. Hence the security context is not available in the authorise
function. So i used the SecurityContextHolder.get() to get the security
context.
But for some of the commands SecurityContextHolder.get() is not working
like prepend and append.
-> When cache create and cache destroy is made, authorise function is
receiving the name as NULL. Why is it so? Because based on the name i am
trying to validate wheather the user is allowed to perform this action
During Sqlline access:
-> authorise function receive the security context as NULL always . So used
the SecurityContextHolder.get() , but still getting NULL. How do i get the
context?
-> While performing create table and drop table, the authorise function
is receiving the name as NULL.
One last question: when the security context is null(during rest call or
sqlline access), can we use the local node context in the authorise
function?
Regards
Radha
