Greetings,
we've been chasing a weird issue in a two node cluster for few days now.
We have a spring boot application bundled with an ignite server node.
We use invokeAsync on TRANSACTIONAL PARTITIONED cache with 1 backup. We
assume that each node in the two node cluster has a copy of the other
node's data. In a way, this mimics REPLICATED cache configuration. Our
business logic is written within an EntryProcessor. The "business code"
in the EntryProcessor is idempotent and arguments to the processor are
fixed. At the end of the "invokeAsync" call, i.e. when IgniteFuture is
resolved, we return a value returned from the EntryProcessor via REST to
the caller of our API.
The problem occurres when one of the two nodes is restarted (triggering
re-balancing) and we simultaneously receive a call to our REST API
launching a businesses computation in EntryProcessor.
The code in EntryProcessor properly computes a new value that we want to
store in the cache. No exception is thrown so we leak it out the REST
caller as a return value, but when rebalancing finishes, the value is
not in the cache anymore.
Yet the caller "saw" and stored the value we returned from our
EntryProcessor.
We did experiment with various cache settings but the problem simply
persists. In fact we initially used REPLICATED cache configuration but
the behavior was pretty much the same.
We have currently settled on a rather extreme configuration, but the
data is still lost during rebalancing from time to time. We are using
Ignite 2.6 and gatling for REST load testing.
The load on the REST api and consequently on Ignite is not very high.
setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)
setCacheMode(CacheMode.PARTITIONED)
setBackups(1)
setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC)
setRebalanceMode(CacheRebalanceMode.SYNC)
setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE)
setAffinity(new RendezvousAffinityFunction().setPartitions(2))
I would appreciate any pointers what may be wrong with our setup/config.
Thank you.
Kamil
