RE: Is ID generator split brain compliant?

[abatra]

It worked. Directory set for IGNITE_HOME was getting cleaned up because it
wasn't on a mount point or docker volume. Once I set IGNITE_HOME to a
directory that persisted between container stop and start, the ID generator
worked as expected.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Is ID generator split brain compliant?

[abatra]

Server logs
  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Is ID generator split brain compliant?

[abatra]

I did not see any locks related logs. 
I did set consistent ID to make sure Ignite looks for the same folder on a
restart. But it did not work. On docker restart i.e. the ID generator starts
generating a sequence from the initial value.

I am attaching the ignite logs for server restart in this message. It's a
two node deployment, 1 server and 1 client where I keep client down all the
time. For that reason, you may see an expected NoRouteException to the
client.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Is ID generator split brain compliant?

[Stanislav Lukyanov]

Hi,

If the issue is that the node is starting with a new persistent storage each 
time
there could some issue with file system.

The algorithm to choose a persistent folder is
1) If IgniteConfiguration.consistentId is specified, use the folder of that name
2) Otherwise, check if there are any existing folders not locked by other 
nodes; if one is found, use that
3) Otherwise, generate a new consistentId and create a new folder for it
I assume that when Ignite checks your folders on step 2 they appear to be 
locked even though they are not.
Perhaps the file system doesn’t unlock them fast enough, or something similar 
happens.
Look for messages like “Unable to acquire lock to file” at the start of the 
nodes to see if that’s the case.

Try to specify consistentId explicitly – perhaps it’ll solve the issue.

Thanks,
Stan

From: abatra
Sent: 1 октября 2018 г. 11:47
To: user@ignite.apache.org
Subject: Re: Is ID generator split brain compliant?

I believe it won't work either. I will try it soon and share the result. 

Could you please share some pointers that I should look at for debugging
persistence? I have org.apache.ignite=DEBUG in logs and IGNITE_QUITE set to
false. There a lot of logs, most of which I can not make sense of w.r.t.
persistence.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[abatra]

I believe it won't work either. I will try it soon and share the result. 

Could you please share some pointers that I should look at for debugging
persistence? I have org.apache.ignite=DEBUG in logs and IGNITE_QUITE set to
false. There a lot of logs, most of which I can not make sense of w.r.t.
persistence.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[Pavel Pereslegin]

Hello Ankit,

I'm not an expert in docker, but for me it looks like you have a problem
with overall cache persistence (not only with AtomicSequence), could you
try to put some values into cache and check their availability after the
restart?


пт, 28 сент. 2018 г. в 23:45, abatra :

> To give more information on how I restart the server:
>
> I run server and client inside their own respective docker container and I
> issue 'docker stop' for server node container ID to stop and then restart
> it.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Is ID generator split brain compliant?

[abatra]

To give more information on how I restart the server:

I run server and client inside their own respective docker container and I
issue 'docker stop' for server node container ID to stop and then restart
it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[abatra]

I tried debugging it a bit. 

Now, I am running only 1 server node. I am running other node as client.
Even with single server node, the atomic sequence is reset to initial value
after the server node restarts.

I checked the IGNITE_HOME folder, on every restart, the contents are
recreated with a different node ID.

node00-46580d24-2fdb-484a-ba63-3a05248c0b8d

*ID after restart*
node00-beae3aab-6165-485a-96e3-1a2fe5b7f20b

I believe this might be the reason for the sequence to not persist across
reboots. However, I am unsure why am I running into this because I am pretty
sure that I am running only 1 server from 1 node.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[Pavel Pereslegin]

Ankit,

 > 1. I run two nodes in the cluster.
I checked 2 nodes cluster restart (simultaneously, 2 processes on same
machine) with the same result - after restart the "current" value !=
"initial" value.

 > 2. I restart the entire node and hence the server gets restarted.
I gave this example only for clarity - in this case it does not matter
how you stop the server. The main thing is that the persistence
directory should not be cleared, but since you mentioned that the data
in the cache is available after restart - I believe it is not cleaned

 > Otherwise, I am not sure how is that working out for you.
Please, try run code by yourself and check results.

 > I see that you are stressing on "cleaned persistence directory".
It's not necessary - I cleaned persistence directory just to see the
expected result.

Persistence should work fine for atomics "out of the box", otherwise
we should create an issue in ASF jira. But for the time being I can
not understand in what case this issue occurs.
пт, 28 сент. 2018 г. в 10:52, abatra :
>
> There are only two differences in the way you tried the example:
>
> 1. I run two nodes in the cluster.
> 2. I do not stop and configure the server while the JVM is running. I
> restart the entire node and hence the server gets restarted.
>
> Otherwise, I am not sure how is that working out for you.
>
> Also, I see that you are stressing on "cleaned persistence directory".
> Please let me know if I am missing something there.
>
> My server nodes run as two different docker containers on two different open
> stack server nodes.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[abatra]

There are only two differences in the way you tried the example:

1. I run two nodes in the cluster.
2. I do not stop and configure the server while the JVM is running. I
restart the entire node and hence the server gets restarted.

Otherwise, I am not sure how is that working out for you.

Also, I see that you are stressing on "cleaned persistence directory".
Please let me know if I am missing something there.

My server nodes run as two different docker containers on two different open
stack server nodes.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[Pavel Pereslegin]

Ankit,
I  tried test your example with the following code:


configure();

System.out.println("Initial value: " + getCurrentValue("sequence"));

for (int i = 0; i < 10_000; i++)
incrementAndGet("sequence");

stop();

configure();

System.out.println("Value after restart: " + getCurrentValue("sequence"));

stop();


And the output is:

Initial value: 150
Value after restart: 1511000


This is the expected result with cleaned persistence directory
(because on last increment was reserved (+1000 by default) values).
Maybe the problem elsewhere?

чт, 27 сент. 2018 г. в 20:14, abatra :
>
> Configuration
>
> private static final String STORAGE_PATH = 
> "/existingDirectory/ignite/data/storage";
> private static final String WAL_PATH = 
> "/existingDirectory/ignite/data/wal";
> private static final String WAL_ARCHIVE_PATH = 
> "/existingDirectory/ignite/data/wal/archive";
>
> @Override
> public void configure() {
> Ignite ignite = Ignition.start(createConfiguration());
> /*
>  * This call is required if ignite persistence is used.
>  */
> ignite.cluster().active(true);
> mIgniteLogger.info("Ignite cluster is configured.");
> }
>
> @Override
> public void stop() {
> Ignition.stopAll(true);
> }
>
> private IgniteConfiguration createConfiguration() {
> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> igniteConfiguration.setDiscoverySpi(createDiscoverySpi());
> igniteConfiguration.setGridLogger(mIgniteLogger);
> 
> igniteConfiguration.setDataStorageConfiguration(createDataStorageConfiguration());
> return igniteConfiguration;
> }
>
> private DiscoverySpi createDiscoverySpi() {
> checkMinimumClusterIpAddressCount();
> TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new 
> TcpDiscoveryVmIpFinder();
> tcpDiscoveryVmIpFinder.setAddresses(getConfig().getNodeIpAddresses());
> tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);
> return tcpDiscoverySpi;
> }
>
> private DataStorageConfiguration createDataStorageConfiguration() {
> DataStorageConfiguration dataStorageConfiguration = new 
> DataStorageConfiguration();
> 
> dataStorageConfiguration.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> dataStorageConfiguration.setStoragePath(STORAGE_PATH);
> dataStorageConfiguration.setWalPath(WAL_PATH);
> dataStorageConfiguration.setWalArchivePath(WAL_ARCHIVE_PATH);
> return dataStorageConfiguration;
> }
>
> Usage
>
> @Override
> public long getCurrentValue(String key) {
> return getAtomicSequence(key).get();
> }
>
> @Override
> public long incrementAndGet(String key) {
> return incrementAndGet(getAtomicSequence(key));
> }
>
> @Override
> public List incrementAndGet(String key, int count) {
> List result = new ArrayList<>();
> IgniteAtomicSequence atomicSequence = getAtomicSequence(key);
> for (int i = 0; i < count; i++) {
> result.add(incrementAndGet(atomicSequence));
> }
> return result;
> }
>
> private long incrementAndGet(IgniteAtomicSequence igniteAtomicSequence) {
> return igniteAtomicSequence.incrementAndGet();
> }
>
> private IgniteAtomicSequence getAtomicSequence(String name) {
> return Ignition.ignite().atomicSequence(name, 150, true);
> }
>
> Please let me know if more details are required.
> 
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Is ID generator split brain compliant?

[abatra]

*Configuration*
private static final String STORAGE_PATH =
"/existingDirectory/ignite/data/storage";private static final String
WAL_PATH = "/existingDirectory/ignite/data/wal";private static final
String WAL_ARCHIVE_PATH = "/existingDirectory/ignite/data/wal/archive";   
@Overridepublic void configure() {Ignite ignite =
Ignition.start(createConfiguration());/* * This call is
required if ignite persistence is used. */   
ignite.cluster().active(true);mIgniteLogger.info("Ignite cluster is
configured.");}@Overridepublic void stop() {   
Ignition.stopAll(true);}private IgniteConfiguration
createConfiguration() {IgniteConfiguration igniteConfiguration = new
IgniteConfiguration();   
igniteConfiguration.setDiscoverySpi(createDiscoverySpi());   
igniteConfiguration.setGridLogger(mIgniteLogger);   
igniteConfiguration.setDataStorageConfiguration(createDataStorageConfiguration());
   
return igniteConfiguration;}private DiscoverySpi
createDiscoverySpi() {checkMinimumClusterIpAddressCount();   
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();   
TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new
TcpDiscoveryVmIpFinder();   
tcpDiscoveryVmIpFinder.setAddresses(getConfig().getNodeIpAddresses());   
tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);return
tcpDiscoverySpi;}private DataStorageConfiguration
createDataStorageConfiguration() {DataStorageConfiguration
dataStorageConfiguration = new DataStorageConfiguration();   
dataStorageConfiguration.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
   
dataStorageConfiguration.setStoragePath(STORAGE_PATH);   
dataStorageConfiguration.setWalPath(WAL_PATH);   
dataStorageConfiguration.setWalArchivePath(WAL_ARCHIVE_PATH);return
dataStorageConfiguration;}
*Usage*
@Overridepublic long getCurrentValue(String key) {return
getAtomicSequence(key).get();}@Overridepublic long
incrementAndGet(String key) {return
incrementAndGet(getAtomicSequence(key));}@Overridepublic List
incrementAndGet(String key, int count) {List result = new
ArrayList<>();IgniteAtomicSequence atomicSequence =
getAtomicSequence(key);for (int i = 0; i < count; i++) {   
result.add(incrementAndGet(atomicSequence));}return result;   
}private long incrementAndGet(IgniteAtomicSequence igniteAtomicSequence)
{return igniteAtomicSequence.incrementAndGet();}private
IgniteAtomicSequence getAtomicSequence(String name) {return
Ignition.ignite().atomicSequence(name, 150, true);}
Please let me know if more details are required.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[Pavel Pereslegin]

Ankit,
I checked sequence and atomic on Ignite 2.6 with the following code:


try (Ignite ignite = Ignition.start("example-persistent-store.xml")) {
ignite.active(true);

IgniteAtomicSequence seq = ignite.atomicSequence("sequence", 0, true);
IgniteAtomicLong atomic = ignite.atomicLong("long", 0, true);

for (int i = 0; i < 100; i++) {
seq.incrementAndGet();
atomic.incrementAndGet();
}
}

try (Ignite ignite = Ignition.start("example-persistent-store.xml")) {
ignite.active(true);

IgniteAtomicSequence seq = ignite.atomicSequence("sequence", 0, true);
IgniteAtomicLong atomic = ignite.atomicLong("long", 0, true);

assert seq.get() == 1000 : seq.get();
assert atomic.get() == 100 : atomic.get();
}


Code works fine with enabled assertions (-ea) and cleaned persistence
directory. We have sequence value = 1000, atomic value = 100 after
node restart.
Could you tell us in more detail how you use "Ignite ID generator"?

чт, 27 сент. 2018 г. в 16:27, abatra :
>
> Hi Anton,
>
> With persistence enabled, if I reboot the nodes in the cluster, ID generator
> starts generating ID from 1 after the nodes are back up. My expectation is
> that that the latest generated ID(s) will be persisted across cluster reboot
> and the ID generator will start from the latest persisted ID.
>
> I am using the latest version of Ignite i.e. 2.6.0.
>
> -Ankit
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[abatra]

Hi Anton,

With persistence enabled, if I reboot the nodes in the cluster, ID generator
starts generating ID from 1 after the nodes are back up. My expectation is
that that the latest generated ID(s) will be persisted across cluster reboot
and the ID generator will start from the latest persisted ID.

I am using the latest version of Ignite i.e. 2.6.0.

-Ankit



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[Pavel Pereslegin]

Anton,
IgniteAtomicSequence should work fine with persistence.

Ankit,
what issue did you encounter when using IgniteAtomicSequence with enabled
persistence? What version of Apache Ignite you are using?


вт, 25 сент. 2018 г. в 15:11, Anton Vinogradov :

> Denis,
>
> As far as I understand, question is about IgniteAtomicSequence?
> We fixed IgniteSet to be persisted and recovered properly.
>
> Pavel Pereslegin,
>
> Could you please check whether we have the same issue with
> IgniteAtomicSequence?
>
> сб, 22 сент. 2018 г. в 4:17, Denis Magda :
>
>> So far, it looks pretty good except that it does not provide persistence
>>> out
>>> of the box. But I can work around it by backing latest generated ID in a
>>> persistent cache and initializing ID generator with the latest value on a
>>> cluster restart.
>>
>>
>> Sounds like a good solution. *Anton*, I do remember a discussion on the
>> dev list in regards persistence support for data structures. Are we
>> releasing anything related soon? Can't recall all the details.
>>
>> However, one thing I could not find an answer for is if the out of the box
>>> ID generator is split brain compliant. I cannot afford to have a
>>> duplicate
>>> ID and want to understand if duplicate ID(s) could occur in a split-brain
>>> scenario. If yes, what is the recommended approach to handling that
>>> scenario?
>>
>>
>> It should be split-brain tolerant if ZooKeeper Discovery is used:
>>
>> https://apacheignite.readme.io/docs/zookeeper-discovery#section-failures-and-split-brain-handling
>>
>> --
>> Denis
>>
>> On Wed, Sep 19, 2018 at 3:37 PM abatra  wrote:
>>
>>> Hi,
>>>
>>> I have a requirement to create a distributed cluster-unique ID generator
>>> microservice. I have done a PoC on it using Apache Ignite ID Generator.
>>>
>>> I created a 2 node cluster with two instances of microservices running on
>>> each node. Nodes are in the same datacenter (in fact in the same network
>>> and
>>> will always be deployed in the same network) and I use TCP/IP discovery
>>> to
>>> discover cluster nodes.
>>>
>>> So far, it looks pretty good except that it does not provide persistence
>>> out
>>> of the box. But I can work around it by backing latest generated ID in a
>>> persistent cache and initializing ID generator with the latest value on a
>>> cluster restart.
>>>
>>> However, one thing I could not find an answer for is if the out of the
>>> box
>>> ID generator is split brain compliant. I cannot afford to have a
>>> duplicate
>>> ID and want to understand if duplicate ID(s) could occur in a split-brain
>>> scenario. If yes, what is the recommended approach to handling that
>>> scenario?
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: Is ID generator split brain compliant?

[Anton Vinogradov]

Denis,

As far as I understand, question is about IgniteAtomicSequence?
We fixed IgniteSet to be persisted and recovered properly.

Pavel Pereslegin,

Could you please check whether we have the same issue with
IgniteAtomicSequence?

сб, 22 сент. 2018 г. в 4:17, Denis Magda :

> So far, it looks pretty good except that it does not provide persistence
>> out
>> of the box. But I can work around it by backing latest generated ID in a
>> persistent cache and initializing ID generator with the latest value on a
>> cluster restart.
>
>
> Sounds like a good solution. *Anton*, I do remember a discussion on the
> dev list in regards persistence support for data structures. Are we
> releasing anything related soon? Can't recall all the details.
>
> However, one thing I could not find an answer for is if the out of the box
>> ID generator is split brain compliant. I cannot afford to have a duplicate
>> ID and want to understand if duplicate ID(s) could occur in a split-brain
>> scenario. If yes, what is the recommended approach to handling that
>> scenario?
>
>
> It should be split-brain tolerant if ZooKeeper Discovery is used:
>
> https://apacheignite.readme.io/docs/zookeeper-discovery#section-failures-and-split-brain-handling
>
> --
> Denis
>
> On Wed, Sep 19, 2018 at 3:37 PM abatra  wrote:
>
>> Hi,
>>
>> I have a requirement to create a distributed cluster-unique ID generator
>> microservice. I have done a PoC on it using Apache Ignite ID Generator.
>>
>> I created a 2 node cluster with two instances of microservices running on
>> each node. Nodes are in the same datacenter (in fact in the same network
>> and
>> will always be deployed in the same network) and I use TCP/IP discovery to
>> discover cluster nodes.
>>
>> So far, it looks pretty good except that it does not provide persistence
>> out
>> of the box. But I can work around it by backing latest generated ID in a
>> persistent cache and initializing ID generator with the latest value on a
>> cluster restart.
>>
>> However, one thing I could not find an answer for is if the out of the box
>> ID generator is split brain compliant. I cannot afford to have a duplicate
>> ID and want to understand if duplicate ID(s) could occur in a split-brain
>> scenario. If yes, what is the recommended approach to handling that
>> scenario?
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Is ID generator split brain compliant?

[Denis Magda]

>
> So far, it looks pretty good except that it does not provide persistence
> out
> of the box. But I can work around it by backing latest generated ID in a
> persistent cache and initializing ID generator with the latest value on a
> cluster restart.


Sounds like a good solution. *Anton*, I do remember a discussion on the dev
list in regards persistence support for data structures. Are we releasing
anything related soon? Can't recall all the details.

However, one thing I could not find an answer for is if the out of the box
> ID generator is split brain compliant. I cannot afford to have a duplicate
> ID and want to understand if duplicate ID(s) could occur in a split-brain
> scenario. If yes, what is the recommended approach to handling that
> scenario?


It should be split-brain tolerant if ZooKeeper Discovery is used:
https://apacheignite.readme.io/docs/zookeeper-discovery#section-failures-and-split-brain-handling

--
Denis

On Wed, Sep 19, 2018 at 3:37 PM abatra  wrote:

> Hi,
>
> I have a requirement to create a distributed cluster-unique ID generator
> microservice. I have done a PoC on it using Apache Ignite ID Generator.
>
> I created a 2 node cluster with two instances of microservices running on
> each node. Nodes are in the same datacenter (in fact in the same network
> and
> will always be deployed in the same network) and I use TCP/IP discovery to
> discover cluster nodes.
>
> So far, it looks pretty good except that it does not provide persistence
> out
> of the box. But I can work around it by backing latest generated ID in a
> persistent cache and initializing ID generator with the latest value on a
> cluster restart.
>
> However, one thing I could not find an answer for is if the out of the box
> ID generator is split brain compliant. I cannot afford to have a duplicate
> ID and want to understand if duplicate ID(s) could occur in a split-brain
> scenario. If yes, what is the recommended approach to handling that
> scenario?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Is ID generator split brain compliant?

[Jörn Franke]

I think you need to also look at the processes that are using the id in case of 
a split brain scenario.
A unique identifier is always some centralistic approach either it is done by 
one central service or a central rule that is enforced in a distributed fashion.

For instance, in your case you can have x nodes and each of them generates a 
unique id starting with nodename_localuniqueid of the node. In this case you 
can be also sure in case of a splitbrain scenario that the ids are unique if 
and only if the node names are unique.

> On 19. Sep 2018, at 21:36, abatra  wrote:
> 
> Hi, 
> 
> I have a requirement to create a distributed cluster-unique ID generator
> microservice. I have done a PoC on it using Apache Ignite ID Generator. 
> 
> I created a 2 node cluster with two instances of microservices running on
> each node. Nodes are in the same datacenter (in fact in the same network and
> will always be deployed in the same network) and I use TCP/IP discovery to
> discover cluster nodes. 
> 
> So far, it looks pretty good except that it does not provide persistence out
> of the box. But I can work around it by backing latest generated ID in a
> persistent cache and initializing ID generator with the latest value on a
> cluster restart. 
> 
> However, one thing I could not find an answer for is if the out of the box
> ID generator is split brain compliant. I cannot afford to have a duplicate
> ID and want to understand if duplicate ID(s) could occur in a split-brain
> scenario. If yes, what is the recommended approach to handling that
> scenario?
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/