[
https://issues.apache.org/jira/browse/IGNITE-29241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140873#comment-18140873
]
Reza Taroosheh commented on IGNITE-29241:
-----------------------------------------
Update on the investigation. The draft PR now has more tests
([c577ca96c2|https://github.com/apache/ignite/pull/13779/commits/c577ca96c2bd37db442d356ec1934cca27963bb8]).
No fix has been pushed yet.
*Cause.* An {{Externalizable}} value is marshalled inside
{{BPlusTree.invoke()}}. If its class isn't registered yet, the thread waits
there for registration. An interrupt ends the wait with a
{{BinaryObjectException}}, and {{BPlusTree.invoke()}} reports that
{{RuntimeException}} as {{CorruptedTreeException}}.
Each such class needs two registrations: its binary type ID, and its optimized
marshaller ID. The second one is done in
{{OptimizedMarshallerUtils.classDescriptor()}}, which always waits.
*put, putAll and put to a remote primary.* The value exists before the tree, so
its bytes can be made before the tree and before the entry locks. A failed
registration then fails the whole request before any key is written. With this
change in my local copy, the tests for these paths pass.
*invoke and the interceptor.* The new value does not exist before the tree,
because it is computed from the old value, which {{AtomicCacheUpdateClosure}}
gets inside {{BPlusTree.invoke()}}. The existing {{failIfUnregistered}} retry
does not cover the optimized marshaller. With the local change above, two tests
still fail with {{CorruptedTreeException}}:
* {{testInterruptedInvokeOfExternalizableClassWithRegisteredBinaryType}}: only
the optimized marshaller ID is left to register.
* {{testInterruptedInvokeOfNestedUnregisteredExternalizableClass}}: the
unregistered class is written inside {{writeExternal()}}.
The interceptor and read-through also make their values in
{{AtomicCacheUpdateClosure}}, so I expect the same problem there, but I have no
failing test for them yet.
*Next.* I am trying to run the entry processor before the tree, as
{{GridDhtAtomicCache.updateWithBatch()}} and {{TRANSACTIONAL}} caches already
do, so that no new value is marshaled inside the tree. The cost is one more key
read for each {{invoke}}.
> An interrupted ATOMIC put of a new Externalizable class is reported as a
> corrupted B+Tree and stops the node
> ------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-29241
> URL: https://issues.apache.org/jira/browse/IGNITE-29241
> Project: Ignite
> Issue Type: Bug
> Reporter: Reza Taroosheh
> Assignee: Reza Taroosheh
> Priority: Major
> Fix For: 2.20
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a thread is interrupted while an {{ATOMIC}} cache update writes a value of
> an {{Externalizable}} class (or a class with its own
> {{writeObject}}/{{readObject}}) that the node has not used yet, the update
> fails with {{CorruptedTreeException: B+Tree is corrupted}}. This is a
> {{CRITICAL_ERROR}}, and the default {{StopNodeOrHaltFailureHandler}} halts
> the JVM.
> Nothing is corrupted: the interrupt only ends the wait for the class name
> registration. An interrupt is an ordinary event on a pooled thread, for
> example at {{ExecutorService.shutdownNow()}}. It should fail the operation,
> not stop the node.
> Steps:
> # Start a server node with an {{ATOMIC}} cache.
> # Interrupt a thread, then call {{cache.put(1, value)}} from it, where
> {{value}} is of an {{Externalizable}} class not used in the cluster before.
> Actual: the put fails, and the failure handler is called with
> {{CorruptedTreeException: B+Tree is corrupted [... msg=Runtime failure on
> search row ...]}}, caused by {{BinaryObjectException: Failed to register
> class}}, caused by {{IgniteInterruptedCheckedException: Got interrupted while
> waiting for future to complete}}.
> Expected: the put fails with an exception, the failure handler is not called,
> and the node keeps running.
> The same interrupt fails the operation cleanly, without a failure handler
> call, when the value's class is handled by the binary marshaller, when the
> {{Externalizable}} class is nested in a binary value, or when the cache is
> {{TRANSACTIONAL}}, and the write is in a transaction.
> Workaround: write each such class once, on a thread that is not interrupted,
> before any thread that may be interrupted writes it.
> {{IgniteBinary.registerClass}} and {{IgniteBinary.toBinary}} do not help.
> Reproduced on 2.18.0 (reported on the dev list:
> https://lists.apache.org/thread.html/qqyp941vb5vmxlzz0d18yglzjyrw3t7r), and
> on master by a new test.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
