siddharthteotia commented on PR #14806:
URL: https://github.com/apache/pinot/pull/14806#issuecomment-2594712114

   I think a better approach would be to rethink how we are propagating the 
execution state during the course of query execution. This would be beneficial 
for debugging  and in fact live analysis too.
   
   Imagine a  a JOIN + aggregation query that runs for 10secs. As the data is 
flowing upstream, a tool can be built to look into the live-progress of the 
query. Built on top of vanilla explain plan, this tool will show the 
multi-stage plan, which executors are running which stage and the progress of 
the stage at the operator level. 
   
   At any point in time (with some lag), we should be able to drill down into 
whether a server is waiting on input, if received then how many records 
received from downstream, how many records produced for upstream so on and so 
forth. 
   
   This will likely require change to the protocol but more importantly will 
require how the state is measured and communicated.
   
   I don't want to be anal about it but in general, there is no point in 
emitting a metric which we know may not be always correct to begin with. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to