bziobrowski commented on code in PR #14662:
URL: https://github.com/apache/pinot/pull/14662#discussion_r1886664452
##########
pinot-core/src/main/java/org/apache/pinot/core/data/table/IndexedTable.java:
##########
@@ -157,11 +166,44 @@ public void finish(boolean sort, boolean
storeFinalResult) {
for (int i = 0; i < numAggregationFunctions; i++) {
columnDataTypes[i + _numKeyColumns] =
_aggregationFunctions[i].getFinalResultColumnType();
}
- for (Record record : _topRecords) {
- Object[] values = record.getValues();
- for (int i = 0; i < numAggregationFunctions; i++) {
- int colId = i + _numKeyColumns;
- values[colId] =
_aggregationFunctions[i].extractFinalResult(values[colId]);
Review Comment:
I think it'd make sense to either :
- put an upper limit on _numThreadsForFinalReduce (e.g. 2 or 3*
Runtime.getRuntime().availableProcessors()) or
- change the variable to a boolean flag `enableParallelFinalReduce` and use
a sensible number of task
to prevent using excessive number of futures or various error modes, e.g.
if _numThreadsForFinalReduce is Integer.MAX_VALUE then chunkSize is going
to be negative.
If shared thread pool is overwhelmed by running tasks it might be good to
use current thread not only to wait but also task processing, stealing tasks
until there's nothing left and only then waiting for futures to finish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
