Intermittent java.util.ConcurrentModificationException and java.lang.NullPointerException on TomEE 7.0.2

[Danilo Cominotti Marques]

Hello there,

I am testing a JAX-RS + CDI + EJB app on TomEE Plus 7.0.2 and am getting
the following exception from time to time:


Exception in thread "managed-thread-88"
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
        at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
        at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
        at java.util.HashMap.putMapEntries(HashMap.java:511)
        at java.util.HashMap.putAll(HashMap.java:784)
        at
java.util.Collections$SynchronizedMap.putAll(Collections.java:2594)
        at
org.apache.openejb.core.ThreadContext.<init>(ThreadContext.java:143)
        at
org.apache.openejb.threads.task.CUTask$Context.enter(CUTask.java:188)
        at org.apache.openejb.threads.task.CUTask.invoke(CUTask.java:78)
        at
org.apache.openejb.threads.task.CURunnable.run(CURunnable.java:32)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


This issue shows up intermittently during my tests with a Gatling script,
which simulates 1000 users sending 10 requests each as fast as possible.
Here is a code snippet from my EJB Session Bean:


@Stateless
public class AsyncFeatureRunner implements FeatureRunner {

    @Resource
    private ManagedExecutorService managedExecutorService;


    @Override
    public <Input, Output> CompletableFuture<Output>
run(RunnableFeature<Input, Output> runnableFeature, Input featureInput) {
        return CompletableFuture.supplyAsync(
                () -> runnableFeature.run(featureInput),
managedExecutorService
        );
    }
}


Also, some of my HTTP requests are timing out/being refused, although my
NIO2 connector's configuration seems OK. Here is the error message I get
when I stop TomEE after I notice my Gatling script is taking too long to
finish:


06-Feb-2017 14:00:08.057 SEVERE [http-nio2-8080-exec-142]
sun.reflect.NativeMethodAccessorImpl.invoke Exception while processing an
asynchronous request
 java.lang.NullPointerException
        at
org.apache.catalina.core.AsyncContextImpl.setErrorState(AsyncContextImpl.java:412)
        at
org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:158)
        at
org.apache.coyote.AbstractProcessor.dispatch(AbstractProcessor.java:228)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
        at
org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1609)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)


Here is the connector's configuration from server.xml:


<Connector port="8080" redirectPort="8443"
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
           maxThreads="200" maxConnections="10000" acceptCount="10000"
SSLEnabled="false">
</Connector>


and here is my JAX-RS resource's code:


@Path("/test/")
@RequestScoped
public RestEndpoint {

    @Inject
    private FeatureRunner featureRunner;

    @Inject
    private CreateUserFeature createUserFeature;


    @GET
    @Path("/")
    @Produces("application/json")
    public void post(@Suspended final AsyncResponse asyncResponse) {
        featureRunner.run(
                createUserFeature, new CreateUserFeature.Input()
        ).thenAccept(
                output -> asyncResponse.resume(
                        Response.ok().build()
                )
        ).exceptionally(
                throwable -> {
                    asyncResponse.resume(
                            Response.serverError().build()
                    );
                    return null;
                }
        );
    }
}


Finally, here is the Gatling script:


import scala.concurrent.duration._

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._

class RecordedSimulation extends Simulation {

val httpProtocol = http
.baseURL("http://localhost:8080";)
.inferHtmlResources()
.acceptHeader("text/html,application/json,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
.acceptEncodingHeader("gzip, deflate, sdch")
.acceptLanguageHeader("en-US,en;q=0.8,pt-BR;q=0.6,pt;q=0.4")
.upgradeInsecureRequestsHeader("1")
.userAgentHeader("Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36")



    val uri1 = "http://localhost:8080/my-app/rest/test";

val scn = scenario("RecordedSimulation").repeat(10, "n") {
exec(http("request_0")
.get("/my-app/rest/test"))
}

setUp(scn.inject(atOnceUsers(1000))).protocols(httpProtocol)
}


Does anyone know what might be the cause of these issues?

Thank you.

Regards,

Danilo Cominotti Marques