richardstartin opened a new issue #7502:
URL: https://github.com/apache/pinot/issues/7502
There are currently a few bugs in the tracing implementation:
1. `TraceContext` accessed concurrently from different threads leading to
`ConcurrentModificationException`
```
2021/09/30 17:47:38.875 ERROR [QueryScheduler] [pqr-12] Encountered
exception while processing requestId 7 from broker Broker_192.168.0.22_8000
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
~[?:?]
at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
at
org.apache.pinot.core.util.trace.TraceContext$Trace.toJson(TraceContext.java:90)
~[classes/:?]
at
org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:189)
~[classes/:?]
at
org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:236)
~[classes/:?]
at
org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)
~[classes/:?]
at
org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154)
~[classes/:?]
at
org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138)
~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
[guava-20.0.jar:?]
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
[guava-20.0.jar:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
[guava-20.0.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
<img width="1065" alt="Screenshot 2021-09-30 at 17 47 46"
src="https://user-images.githubusercontent.com/16439049/135497563-af35fcba-046b-4fce-aa7d-175cc307b3ea.png">
2. `NullPointerException` in query runner thread pool after the request id
has been cleared
```
2021/09/30 17:47:48.858 ERROR [QueryScheduler] [pqr-15] Encountered
exception while processing requestId 8 from broker Broker_192.168.0.22_8000
java.lang.NullPointerException: null
at
org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:188)
~[classes/:?]
at
org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:236)
~[classes/:?]
at
org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)
~[classes/:?]
at
org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154)
~[classes/:?]
at
org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138)
~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
[guava-20.0.jar:?]
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
[guava-20.0.jar:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
[guava-20.0.jar:?]
```
<img width="1018" alt="Screenshot 2021-09-30 at 17 47 53"
src="https://user-images.githubusercontent.com/16439049/135498035-8a32eba0-e54b-46b4-a979-e6a736c2f81a.png">
3. Tracing errors cause the query to fail (tracing errors should never
impact what's being traced, it should simply be recorded that tracing failed
without changing behaviour.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]