[
https://issues.apache.org/jira/browse/TOMEE-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Zowalla updated TOMEE-4699:
-----------------------------------
Fix Version/s: 11.0.0
10.3.0
> ThreadContext constructor is not thread-safe, causing loss of submitted tasks.
> ------------------------------------------------------------------------------
>
> Key: TOMEE-4699
> URL: https://issues.apache.org/jira/browse/TOMEE-4699
> Project: TomEE
> Issue Type: Bug
> Components: TomEE Core Server
> Affects Versions: 10.2.0
> Reporter: Jorren Hendriks
> Priority: Major
> Fix For: 11.0.0, 10.3.0
>
> Attachments: stacktrace.txt
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Some of our tomee applications have experienced issues losing tasks submitted
> to a managed executor.
> The root cause has been identified as a
> `{_}ConcurrentModificationException{_}` in the ThreadContext constructor
> (stacktrace attached).
> {code:java}
> public ThreadContext(final ThreadContext that) {
> this.beanContext = that.beanContext;
> this.primaryKey = that.primaryKey;
> this.data.putAll(that.data); // throws ConcurrentModificationException
> when that.data is modified on another thread
> this.oldClassLoader = that.oldClassLoader;
> }{code}
> `{_}HashMap.putAll{_}` will iterate the entrySet, which is not thread-safe on
> the `{_}Collections.synchronizedMap{_}` wrapped HashMap currently in use.
> A possible solution would be to use a thread-safe Map implementation.
> This constructor is called when entering the snapshot in the
> [{_}ApplicationThreadContextProvider{_}|[https://github.com/apache/tomee/blob/main/container/openejb-core/src/main/java/org/apache/openejb/threads/impl/ApplicationThreadContextProvider.java#L76|https://github.com/apache/tomee/blob/main/container/openejb-core/src/main/java/org/apache/openejb/threads/impl/ApplicationThreadContextProvider.java#L76].]]
> This issue also revealed possible other issues for us:
> 1. `{_}CUTask{_}` does not handle exceptions before starting the task. Any
> exceptions in `{_}contextService.enter(snapshot){_}` cancel invocation, but
> do not abort the task.
> 2. Exceptions in a `{_}CUTask{_}`/`{_}CURunnable{_}` are propagated to the
> `{_}ThreadPoolExecutor{_}`. The current implementation does not expose these
> exceptions which makes this a silent failure.
> If you'd prefer that I can file these as separate issues, but they are
> related to this problem.
> h2. Reproduction
> This is a race condition which very rarely occurs. We were able to reproduce
> it (under high load) within an hour on one of our applications with a
> debugger attached. So far we were not able to create a minimal reproduction
> outside our application. I will update the issue when we do.
> The pattern we use in the application is as follows, the comments indicate my
> interpretation of what happens:
> {code:java}
> // MyTaskExecutor.java
> @Stateless
> public class MyTaskExecutor {
> @Resource
> private ManagedExecutorService managedExecutorService; // default TomEE
> managed executor service
> @Override
> public void execute(@NotNull Runnable runnable) {
> // 1. runnable is wrapped in CUTask. ThreadContext is copied on the
> executor thread.
> managedExecutorService.submit(runnable);
> } // 2. method returns immediately, causing an update to the current
> ThreadContext.
> } {code}
> When the exception occurs, 1 & 2 both execute at the same time, accessing the
> current ThreadContext concurrently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
