jasperjiaguo commented on code in PR #11200:
URL: https://github.com/apache/pinot/pull/11200#discussion_r1286251914
##########
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/BaseDistinctAggregateAggregationFunction.java:
##########
@@ -112,6 +113,9 @@ public Set merge(Set intermediateResult1, Set
intermediateResult2) {
if (intermediateResult2.isEmpty()) {
return intermediateResult1;
}
+
+ Tracing.ThreadAccountantOps.sampleAndCheckInterruption();
Review Comment:
Good question. The complete stack trace is
```
Thread name: "brw-14", daemon: false
java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
it.unimi.dsi.fastutil.longs.LongOpenHashSet.rehash(LongOpenHashSet.java:592)
Local variables
it.unimi.dsi.fastutil.longs.LongOpenHashSet.tryCapacity(LongOpenHashSet.java:266)
it.unimi.dsi.fastutil.longs.LongOpenHashSet.addAll(LongOpenHashSet.java:283)
Local variables
org.apache.pinot.core.query.aggregation.function.BaseDistinctAggregateAggregationFunction.merge(BaseDistinctAggregateAggregationFunction.java:115)
Local variables
org.apache.pinot.core.query.aggregation.function.BaseDistinctAggregateAggregationFunction.merge(BaseDistinctAggregateAggregationFunction.java:47)
org.apache.pinot.core.data.table.IndexedTable.lambda$addOrUpdateRecord$0(IndexedTable.java:113)
Local variables
org.apache.pinot.core.data.table.IndexedTable$$Lambda$1890.apply(, line not
available)
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1932)
Local variables
org.apache.pinot.core.data.table.IndexedTable.addOrUpdateRecord(IndexedTable.java:105)
org.apache.pinot.core.data.table.ConcurrentIndexedTable.upsertWithoutOrderBy(ConcurrentIndexedTable.java:77)
```
So this BaseDistinctAggregateAggregationFunction.merge function is called in
IndexedTable. addOrUpdateRecord in an iterative manner, therefore, we can call
this once in merge and it will get updated multiple times in the outter loop.
The reason we cannot add more precise tracing is we cannot break set's
addAll function and inject instrumentation code further inside it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
