Greetings,
while testing our Apache Ignite based app with Gatling, I have noticed
that sometimes a write to a single cache fails, because it has different
writeBehind setting than some other cache. The problem is that the two
caches are not part of the same transaction and should have been updated
in two different transactions, not one.
I do not have a reproducer at hand but the scenario is simple:
* Have server node bundled with your application
* Have at least two nodes
* Have two caches with PARTITIONED and TRANSACTIONAL
* Have one of the caches with writeBehind=true and the other one with
writeBehind=false
* Use optimistic transaction with the first cache (SERIALIZED +
OPTIMISTIC)
* Update both caches in sequence with two separate transactions but
demarcate the first transaction with commitAsync() instead of commit()
* Have non-Ignite fixed size thread pool initiating the commits
* Shoot your application down with Gatling with no pauses or delays to
stress the whole system
After some time you may get "Failed to enlist new cache to existing
transaction" errors for operations that should not have been committed
as a single transaction since the transaction boundary was demarcated
with commitAsync().
I've tried to narrow down the issue to these sections:
https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxStateImpl.java#L266-L271
https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTransactionalCacheAdapter.java#L1112
My guess is that IgniteTxStateImpl is somehow thread local/or cached and
under heavy load in a fixed sized thread pool, separate transactions are
merged into one "batch" for optimization. The trouble is that
commitAsync() in this case does not demarcate your transactions as you
would have guessed.
Is this the expected behavior?
--
S pozdravom,
Kamil Mišúth
