richardstartin opened a new issue #7502:
URL: https://github.com/apache/pinot/issues/7502


   There are currently a few bugs in the tracing implementation:
   
   1. `TraceContext` accessed concurrently from different threads leading to 
`ConcurrentModificationException` 
   ```
   2021/09/30 17:47:38.875 ERROR [QueryScheduler] [pqr-12] Encountered 
exception while processing requestId 7 from broker Broker_192.168.0.22_8000
   java.util.ConcurrentModificationException: null
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) 
~[?:?]
        at java.util.ArrayList$Itr.next(ArrayList.java:997) ~[?:?]
        at 
org.apache.pinot.core.util.trace.TraceContext$Trace.toJson(TraceContext.java:90)
 ~[classes/:?]
        at 
org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:189)
 ~[classes/:?]
        at 
org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:236)
 ~[classes/:?]
        at 
org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)
 ~[classes/:?]
        at 
org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154)
 ~[classes/:?]
        at 
org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138)
 ~[classes/:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
 [guava-20.0.jar:?]
        at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
 [guava-20.0.jar:?]
        at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
 [guava-20.0.jar:?]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
   ```
   <img width="1065" alt="Screenshot 2021-09-30 at 17 47 46" 
src="https://user-images.githubusercontent.com/16439049/135497563-af35fcba-046b-4fce-aa7d-175cc307b3ea.png";>
   
   2. `NullPointerException` in query runner thread pool after the request id 
has been cleared
   ```
   2021/09/30 17:47:48.858 ERROR [QueryScheduler] [pqr-15] Encountered 
exception while processing requestId 8 from broker Broker_192.168.0.22_8000
   java.lang.NullPointerException: null
        at 
org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:188)
 ~[classes/:?]
        at 
org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:236)
 ~[classes/:?]
        at 
org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)
 ~[classes/:?]
        at 
org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:154)
 ~[classes/:?]
        at 
org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:138)
 ~[classes/:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
 [guava-20.0.jar:?]
        at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
 [guava-20.0.jar:?]
        at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
 [guava-20.0.jar:?]
   ```
   <img width="1018" alt="Screenshot 2021-09-30 at 17 47 53" 
src="https://user-images.githubusercontent.com/16439049/135498035-8a32eba0-e54b-46b4-a979-e6a736c2f81a.png";>
   
   3. Tracing errors cause the query to fail (tracing errors should never 
impact what's being traced, it should simply be recorded that tracing failed 
without changing behaviour.)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to