siddharthteotia commented on a change in pull request #6710:
URL: https://github.com/apache/incubator-pinot/pull/6710#discussion_r603708603
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
##########
@@ -161,13 +163,15 @@ public void stop() {
queryRequest.getBrokerId(), e);
// For not handled exceptions
serverMetrics.addMeteredGlobalValue(ServerMeter.UNCAUGHT_EXCEPTIONS, 1);
- dataTable = new DataTableImplV2();
+ dataTable = new DataTableImplV3();
dataTable.addException(QueryException.getException(QueryException.INTERNAL_ERROR,
e));
}
long requestId = queryRequest.getRequestId();
Map<String, String> dataTableMetadata = dataTable.getMetadata();
dataTableMetadata.put(DataTable.REQUEST_ID_METADATA_KEY,
Long.toString(requestId));
+ byte[] responseBytes = serializeDataTable(queryRequest, dataTable);
+
Review comment:
I think it is useful for debugging and understanding how time is being
spent by emitting and logging each cpu time cost metric separately. We should
send an aggregated value to the broker though.
I understand this gets convoluted, but may be this is what we can do
Option 1 (simpler)
- Log, emit (serve table gauge) and send to broker separately. This
simplifies everything. Broker (the follow up PR) will anyway sum up the costs
from each server. It might as well do a per server summation first before
aggregating across servers
Option 2 (somewhat hacky)
If we want to log and emit (Server table gauge) separately but want to send
a single value to broker
- Provide an API on the DataTable to expose the data table serialization
cost.
- The API will be implemented by DataTableImplV3 to return the actual
measured cost and DataTableImplV2 can simply return -1
- Call that API here after line 173 to get the serialization cpu time cost.
Use that to emit the metric to server table gauge and then log it at line 225.
- Subtract the previous value from line 193 and then emit the latter and log
it
I prefer option 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
