Hi Ilya,
I have tracked down this issue to a racy behavior in the business code
and Ignite thread pool starvation caused by the application code.
Sorry for the false alarm.
---
S pozdravom,
Kamil Mišúth
On 2019-05-22 18:46, Ilya Kasnacheev wrote:
Hello!
Do you have reproducer for this behavior? Have you tried the same
scenario on 2.7? I doubt anyone will take effort to debug 2.6.
Regards,
--
Ilya Kasnacheev
чт, 25 апр. 2019 г. в 18:59, kimec.ethome.sk [1]
<[email protected]>:
Greetings,
we've been chasing a weird issue in a two node cluster for few days
now.
We have a spring boot application bundled with an ignite server
node.
We use invokeAsync on TRANSACTIONAL PARTITIONED cache with 1 backup.
We
assume that each node in the two node cluster has a copy of the
other
node's data. In a way, this mimics REPLICATED cache configuration.
Our
business logic is written within an EntryProcessor. The "business
code"
in the EntryProcessor is idempotent and arguments to the processor
are
fixed. At the end of the "invokeAsync" call, i.e. when IgniteFuture
is
resolved, we return a value returned from the EntryProcessor via
REST to
the caller of our API.
The problem occurres when one of the two nodes is restarted
(triggering
re-balancing) and we simultaneously receive a call to our REST API
launching a businesses computation in EntryProcessor.
The code in EntryProcessor properly computes a new value that we
want to
store in the cache. No exception is thrown so we leak it out the
REST
caller as a return value, but when rebalancing finishes, the value
is
not in the cache anymore.
Yet the caller "saw" and stored the value we returned from our
EntryProcessor.
We did experiment with various cache settings but the problem simply
persists. In fact we initially used REPLICATED cache configuration
but
the behavior was pretty much the same.
We have currently settled on a rather extreme configuration, but the
data is still lost during rebalancing from time to time. We are
using
Ignite 2.6 and gatling for REST load testing.
The load on the REST api and consequently on Ignite is not very
high.
setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)
setCacheMode(CacheMode.PARTITIONED)
setBackups(1)
setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC)
setRebalanceMode(CacheRebalanceMode.SYNC)
setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE)
setAffinity(new RendezvousAffinityFunction().setPartitions(2))
I would appreciate any pointers what may be wrong with our
setup/config.
Thank you.
Kamil
Links:
------
[1] http://kimec.ethome.sk
