Re: Node with BaselineTopology cannot join mixed clusterrunningincompatibility mode

[arunkjn]

Thanks Alex,

How can I fix this in the current cluster without resetting the persistence
state of entire cluster?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Node with BaselineTopology cannot join mixed clusterrunningincompatibility mode

[arunkjn]

Hi,

I am experiencing the same issue with a ignite 2.5 cluster deployed in
kubernetes.

Scenario:

5 server nodes hosting some caches. They are deployed multiple times without
issues.
I connected a visor client to this cluster (this is the first time i used
visor ever on this cluster)
On another rolling update, the nodes fail to restart with the message -
class org.apache.ignite.IgniteException: Node with BaselineTopology cannot
join mixed cluster running in compatibility mode
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onGridDataReceived(GridClusterStateProcessor.java:714)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:883)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:1939)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4354)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2744)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)


I can also see this in the logs -

class org.apache.ignite.IgniteCheckedException: Failed to start SPI:
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
marsh=JdkMarshaller
[clsFilter=org.apache.ignite.internal.IgniteKernal$5@3ed03652], reconCnt=10,
reconDelay=2000, maxAckTimeout=60, forceSrvMode=false,
clientReconnectDisabled=false, internalLsnr=null]
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1720)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1033)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693)
at org.apache.ignite.Ignition.start(Ignition.java:352)
at
com.mediaiq.caps.platform.choreography.data.DataNodeStartup.main(DataNodeStartup.java:43)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Thread has been
interrupted.
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:938)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
... 13 more


I am not sure if visor was still connected while deploying, but the problem
seem to persist after disconnecting visor as well. Please advise on how to
fix this. Has this been fixed in 2.6? 

Thanks,
Arun




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Node with BaselineTopology cannot join mixed clusterrunningincompatibility mode

[arunkjn]

Hi,

I am experiencing the same issue with a ignite 2.5 cluster deployed in
kubernetes.

Scenario:

5 server nodes hosting some caches. They are deployed multiple times without
issues.
I connected a visor client to this cluster (this is the first time i used
visor ever on this cluster)
On another rolling update, the nodes fail to restart with the message -
class org.apache.ignite.IgniteException: Node with BaselineTopology cannot
join mixed cluster running in compatibility mode
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onGridDataReceived(GridClusterStateProcessor.java:714)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:883)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:1939)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4354)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2744)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)


I can also see this in the logs -

class org.apache.ignite.IgniteCheckedException: Failed to start SPI:
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
marsh=JdkMarshaller
[clsFilter=org.apache.ignite.internal.IgniteKernal$5@3ed03652], reconCnt=10,
reconDelay=2000, maxAckTimeout=60, forceSrvMode=false,
clientReconnectDisabled=false, internalLsnr=null]
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1720)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1033)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693)
at org.apache.ignite.Ignition.start(Ignition.java:352)
at
com.mediaiq.caps.platform.choreography.data.DataNodeStartup.main(DataNodeStartup.java:43)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Thread has been
interrupted.
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:938)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
... 13 more


I am not sure if visor was still connected while deploying, but the problem
seem to persist after disconnecting visor as well. Please advise on how to
fix this. Has this been fixed in 2.6? 

Thanks,
Arun




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: zombie ignite node

[arunkjn]

Him

It only shows one process each time which is 'grep ignite' the one I am
running. It does not list any other ignite process. I have tried this before
as well.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: zombie ignite node

[arunkjn]

Thanks for the suggestion srecon. I will try that out.

But is there any way I can find out and kill that ignite node? I have tried
deleting the configured wal and persistance directories. Tried JPS to list
all java processes but could not find any. Even rebooted my system but the
zombie node remains there.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

zombie ignite node

[arunkjn]

Hi,

I am using ignite 2.5.0 in local development. When I start a server node in
my code, in the logs I can see that 2 server nodes are present in cluster.

[22:10:46] Topology snapshot [ver=80, servers=2, clients=0, CPUs=8,
offheap=4.6GB, heap=4.0GB]
[22:10:46]   ^-- Node [id=1ADFFFD1-E442-48C6-88A2-C429EC73FB5D,
clusterState=INACTIVE]

with cluster activator tool when I run ./control.sh --activate and set
baseline topology, then running ./cluster.sh --baseline gives me -


Cluster state: active
Current topology version: 80

Baseline nodes:
   
ConsistentID=0:0:0:0:0:0:0:1%lo,127.0.0.1,172.17.0.1,192.168.2.84,192.168.5.192:47500,
STATE=ONLINE
ConsistentID=1, STATE=ONLINE

Number of baseline nodes: 2

The server node that I am running within my code has a consistent ID of 1.
The first node is a zombie node. I am not sure how to stop that zombie node.
Can you please help?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: A series of Apache Ignite meetups in New York

[arunkjn]

Hi Dennis,

Would you please consider writing a blog post on 'Ignite + kubernetes' for
us folks who are not in NY? Also if you can record this session and post
links later, it would be very useful.

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: setting baseline topology in kubernetes

[arunkjn]

Hi Alex,

It accepts an IP and port as an argument. Do I need to enable ignite rest
and expose rest endpoints on cluster nodes for this to work?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

setting baseline topology in kubernetes

[arunkjn]

Hi,

We run a ignite cluster in kubernetes. We currently deploy our containerised
java code which uses xml to configure ignite nodes and start them. We use
Kubernetes Discovery for service discovery.

I want to do automatic cluster activation by defining baseline topology
using control.sh script as described in the docs. I am not able to
understand how to configure control.sh to use kubernetes service discovery
to connect to my running nodes?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Using Persistent ignite queues

[arunkjn]

Hi Alex,

I cannot get it to reproduce in my local environment yet. I will get back to
you with thread dumps as soon as I encounter this issue again.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Using Persistent ignite queues

[arunkjn]

Hi,

I have a scenario where I use a persistent ignite queue across cluster
nodes. 

This is my setup using ignite 2.5-

Few server nodes which hosts some caches in persistance mode and a cluster
singleton ignite service. This ignite service upon initialization,
initializes two persistent queues.

Rest of the nodes in cluster are client nodes. They either put data in queue
or consume data from queue (depending on their role).  Basically I am using
Ignite queue as a worker queue, one party fills the queue and another party
empties it.

I have observed that sometimes the queue gets stuck, as in the queue.take()
method on one node keeps blocking even if there are items in the queue. This
issue does not go away until I restart the whole cluster from fresh state.

I looked into the /etc/ignite/peristance/1 folder (1 is the consistent node
id for one of the nodes). It only contains folders to store the caches. I
cannot see any reference of queue getting saved. I am missing something
here?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Initialize ignite queue when node is starting

[arunkjn]

Thanks, I meant ignite queue. 

If I have initialized an ignite queue in one node, and I want to just obtain
a reference to it on the second node; there is no API to do that currently
without passing in the entire configuration. Is there something I am
missing?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Initialize ignite queue when node is starting

[arunkjn]

Hi,

I give my cache configuration in xml, which the node uses to initialize the
cache when starting up. I want to similarly do this for initializing ignite
queues.

what would be a substitue of -

  
  

  
  

  
  

  


for ignite queues



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re: Nodes can not join the cluster after reboot

[arunkjn]

Hi Evgenii,

I am experiencing the same issue with grid affinity topology.

We have a topology where we have 3 data nodes, 2 service nodes and several
client nodes in the cluster. The issue occurs randomly sometimes when
deploying the nodes. 

I wanted to know that the suggested solution of setting
-DIGNITE_EXCHANGE_COMPATIBILITY_VER_1=true, does it have to be done on all
the nodes? or just service nodes?

Regards,
Arun




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Snapshot backup of Ignite native persistance data

[arunkjn]

Hi Ivan,

Thanks for you reply.

I have some follow up questions.
1. What is the behaviour of cluster when it is in
ignite.cluster.active(false) state? Does it throw error when a cache seek
operation is performed? Or does it wait for cluster to become active again? 
2. How can I reliably know if cluster has become inactive before taking
backup? Is cluster.active(false) a synchronous operation?
3. What do you mean by "consistent ID of your nodes"? Can you please
elaborate?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Snapshot backup of Ignite native persistance data

[arunkjn]

Hi,

I have an ignite cluster which has some durable caches. These caches are
using amazon EBS volumes. I want to backup each EBS node daily for disaster
recovery since we cannot afford losing the data and want disaster recovery
capability.

I was looking around to see if ignite offers a native way of taking reliable
snapshots. I could see its support in the grid gain
distribution(https://www.gridgain.com/products/software/ultimate-edition/cluster-snapshots).
Is there anything available in ignite to do the same?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Stale class on ignite data node

[arunkjn]

Hi Guys,

I have an application in which I segregate ignite nodes into data and
service nodes.

The data node is responsible for hosting cache and queues. The service nodes
host an ignite service which uses cache defined in data nodes. These nodes
are separated because during my application development lifecycle I want to
redeploy service nodes containing new code with each release but I want to
keep the data nodes the same.

I am experiencing an issue where the data node has a stale version of an
entity class in its classpath and is thus not able to execute a service
method. I have to redeploy the data node with updated class for it to work.
What are the suggested ways of getting around this problem? Should I deploy
data nodes when an entity structure changes? What is the downside to doing
this?  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite service method cannot invoke for third time

[arunkjn]

Hi Dmitry,

Sorry this is off the topic but I am a beginner in Java programming. I want
to understand why it helped to make EntryProcessor a static inner class.
Also want to understand how to analyze thread dumps and what information we
can extract from it. If you can point me to some good resource that would be
great.

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite service method cannot invoke for third time

[arunkjn]

So I have separated these classes into independent classes now and made them
non static. This is also working well for me.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite service method cannot invoke for third time

[arunkjn]

Hi Dmitry,

Sorry I don't understand what you mean. Do you mean to do this-

static EntryProcessor workflowStartProcessor = new EntryProcessor() {
@Override
public Object process(MutableEntry entry, Object...
arguments) throws EntryProcessorException {
System.out.println("EntryProcessor started");
Workflow w = entry.getValue();

if (!w.getStatus().equals(Constants.WorkflowStatus.STOPPED))
throw new EntryProcessorException("Workflow not in stopped
state");

w.setStatus(Constants.WorkflowStatus.RUNNING);

entry.setValue(w);

return null;
}
};

@Override
public void startWorkflow(Long id) {
System.out.println("trying to start service");


// try to use lock key here
workflowIgniteCache.invoke(id, workflowStartProcessor);

System.out.println("Sending init events");
sendInitEvents(workflowIgniteCache.get(id));
System.out.println("Sent init events");
}

I tried it but the same thing is happening. Could you please point me to
some example where static inner class based EntryProcessor is used?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Use IgniteQueue in client

[arunkjn]

Thanks Alexey.

I think the problem is that Ignite.queue and Ignite.set javadocs mention
that null is an acceptable entry and also suggest that null can be used when
we do not want to create a new set or queue. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Use IgniteQueue in client

[arunkjn]

Hi,

I want to create an IgniteQueue on a server node and get a reference to this
queue on a client node. I get a null pointer exception on client when I do
this.

This has been asked before in a very old thread and is reportedly fixed as
per  this issue   . But I
am getting the same behaviour using both ignite version 2.2 and 2.3 . Can
someone aware of the development in this regard please update me with the
status of this? Is this a reasonable expectation? Or are there other ways to
achieve this?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Can client node use IgniteQueue?

[arunkjn]

I have also tried version 2.3 with the same issue.
Hi,

I am still experiencing this issue with ignite version 2.2.0

Can you please confirm which version has this been fixed in?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Can client node use IgniteQueue?

[arunkjn]

Hi,

I am still experiencing this issue with ignite version 2.2.0

Can you please confirm which version has this been fixed in?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Transactions and IgniteQueue

[arunkjn]

Thanks Evgenii.

If I get a TransactionOptimisticException during the transaction will it be
thrown at the time of tx.commit() ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Transactions and IgniteQueue

[arunkjn]

Hi,

I am using an optimistic transaction with serializable isolation level on a
transactional cache as follows-

while (true){
try(Transaction tx =
ignite.transactions().txStart(TransactionConcurrency.OPTIMISTIC,
TransactionIsolation.SERIALIZABLE)){

// update all keys of workflow run state corresponding
to this event
flowRunStateIds.stream().forEach(flowRunStateId -> {
FlowRunState state =
workflowRunStateIgniteCache.get(flowRunStateId);
try {
state.getFlowRunEvents().put(event);
} catch (InterruptedException e) {
e.printStackTrace();
}
workflowRunStateIgniteCache.put(flowRunStateId,
state);
});

updatedFlows.stream().forEach(workflowFlow ->
updatedFlowsIgniteQueue.put(workflowFlow));

tx.commit();
break;
}
catch (TransactionOptimisticException e){
// todo: emit a monitoring metric TRANSACTIONS_FAILED
here
System.out.println("Transaction failed. Retrying...");
}
}

Inside the transaction I am updating the cache values and also updating an
IgniteQueue updatedFlowsIgniteQueue. If the transaction fails at the time of
commit, the transactional cache will not be updated. Is the same also true
for IgniteQueue? Also, will my queue be locked during the duration of the
transaction?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Understanding service grid

[arunkjn]

Hi,

I have a very basic beginner question. I followed the documentation for
service grid and I cannot understand why I would need a 'service'
abstraction when I can do the same work directly from an ignite client using
an EntryProcessor?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteCache versioning question

[arunkjn]

Hi Slava,

I am using ignite data grid feature with ignite persistance. I have an
IgniteCache to manage a collection of Workflow objects by
their Long id. I deploy data grid with this cache and populate it with few
workflows. Later if I change the structure of Workflow object, and deploy
this cache on the same cluster how will ignite handle the existing Workflow
objects in the cache? 

Alex,

I am not fully aware of the SQL api for ignite. I am going through the docs
now.


Alexander Paschenko wrote
> Hello Arun,
> 
> Are you aware of possibility to dynamically add columns to your table via
> ALTER TABLE ADD COLUMN?
> 
> Regards,
> Alex
> 
> вт, 17 окт. 2017 г. в 19:27, slava.koptilin 

> slava.koptilin@

> :
> 
>> Hi Arun,
>>
>> I don't quite understand your question.
>> Could you please elaborate in more details? What are you trying to
>> achieve?
>> It would be nice if you can share the pseudo code here.
>>
>> Thanks!
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

IgniteCache versioning question

[arunkjn]

Hi,

This is my first post to an Apache user mailing list and I have done
sufficient groundwork before this. Apologies if I am not following some
guidelines.

I wanted to get others opinion on the best way to version an ignite
collection. If my entity evolves during the application lifecycle, does
ignite offer any native way of handling this? As I understand, the ignite
data nodes need specification of this entity class. 

I can think of having a separate cache for the evolved entity. Like
'user_collection_v2'. Is there any better way of doing this?

Thanks,
Arun



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/