walterddr opened a new issue, #10399:
URL: https://github.com/apache/pinot/issues/10399
Currently trace info is recorded on a thread-local context with requestID as
its primary key.
1. When 2 servers are launched on the same JVM such as quickstart runner, it
can collide and cause issue
2. When 2 brokers individually generates requestID long and send to the same
server simultaneously it will also cause contention (although very rare)
3. for multi-stage engine, the same could happen much more frequently b/c
the same requestID with different stageID can be sent to the same server
(addressed in #10390)
log sample
```
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
~[?:?]
at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
at
org.apache.pinot.core.util.trace.TraceContext$Trace.toJson(TraceContext.java:91)
~[classes/:?]
at
org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:195)
~[classes/:?]
at
org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.executeInternal(ServerQueryExecutorV1Impl.java:284)
~[classes/:?]
at
org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.execute(ServerQueryExecutorV1Impl.java:146)
~[classes/:?]
at
org.apache.pinot.core.query.executor.QueryExecutor.execute(QueryExecutor.java:100)
~[classes/:?]
at
org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154)
~[classes/:?]
at
org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:136)
~[classes/:?]
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java) [?:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
