cryptoe commented on code in PR #17581:
URL: https://github.com/apache/druid/pull/17581#discussion_r1911873372
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java:
##########
@@ -329,6 +331,27 @@ public void run(final QueryListener queryListener) throws
Exception
}
// Call onQueryComplete after Closer is fully closed, ensuring no
controller-related processing is ongoing.
queryListener.onQueryComplete(reportPayload);
+
+ long totalProcessedBytes =
reportPayload.getCounters().copyMap().values().stream()
Review Comment:
I think the place where it has moved is correct.
Rather than ingest in the metric name can we rename the matric to
`input/processed/bytes` or something since we would want that metric in msq
selects as well.
Also the msq code might need to be adjusted so that only leaf nodes
contribute to this metric no ? as an equivalent batch ingest with range
partitioning will show less `processed bytes` since the shuffle stage input is
not being counted for. A simple test should be sufficient to rule this out.
Try a query like `replace bar all using select * from extern(http)
partitioned by day clustered by col1` and an equivalent range partitioning
spec for batch ingestion for the same http input source.
cc @kfaraz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
