SabrinaZhaozyf opened a new pull request, #11942:
URL: https://github.com/apache/pinot/pull/11942
When the InstanceResponseBlock is creating the DataTable based on the final
results in the combine operator, that data table creation is hitting OOM (see
examples below). This PR adds instrumentation for this path under the query
killing framework.
Examples from heap dumps
**Selection**
```
java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
java.util.Arrays.copyOf(Arrays.java:3745)
Local variables
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
Local variables
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
Local variables
java.io.OutputStream.write(OutputStream.java:122)
org.apache.pinot.core.common.datatable.BaseDataTableBuilder.finishRow(BaseDataTableBuilder.java:163)
org.apache.pinot.core.query.selection.SelectionOperatorUtils.getDataTableFromRows(SelectionOperatorUtils.java:342)
Local variables
org.apache.pinot.core.operator.blocks.results.SelectionResultsBlock.getDataTable(SelectionResultsBlock.java:85)
org.apache.pinot.core.operator.blocks.InstanceResponseBlock.toDataOnlyDataTable(InstanceResponseBlock.java:128)
org.apache.pinot.core.operator.blocks.InstanceResponseBlock.toDataTable(InstanceResponseBlock.java:121)
Local variables
org.apache.pinot.core.query.scheduler.QueryScheduler.serializeResponse(QueryScheduler.java:337)
```
**GroupBy**
```
java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
java.util.Arrays.copyOf(Arrays.java:3745)
Local variables
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
Local variables
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
Local variables
java.io.OutputStream.write(OutputStream.java:122)
org.apache.pinot.core.common.datatable.BaseDataTableBuilder.setColumn(BaseDataTableBuilder.java:112)
org.apache.pinot.core.operator.blocks.results.GroupByResultsBlock.setDataTableColumn(GroupByResultsBlock.java:262)
org.apache.pinot.core.operator.blocks.results.GroupByResultsBlock.getDataTable(GroupByResultsBlock.java:209)
Local variables
org.apache.pinot.core.operator.blocks.InstanceResponseBlock.toDataOnlyDataTable(InstanceResponseBlock.java:117)
org.apache.pinot.core.operator.blocks.InstanceResponseBlock.toDataTable(InstanceResponseBlock.java:110)
Local variables
org.apache.pinot.core.query.scheduler.QueryScheduler.serializeResponse(QueryScheduler.java:337)
Local variables
```
**Testing**
- Tests added to`ResourceManagerAccountingTest` verify that memory usage is
sampled, interruption is checked, and `EarlyTerminationException` is thrown in
`getDataTable` by manually setting up the accountant **after** rows/result
blocks are created (only samples for data table creation). Without
instrumentation, tests will throw OOM exception.
- Deterministic integration tests to check that queries are killed exactly
at data table creation can be hard. Reasons being that triggering threshold too
low can cause queries to be killed at combine level before hitting data table
creation while a higher threshold can only kill the query on the server
nondeterministically.
Below is an example of query exception in the response when a kill happened
exactly at data table creation. While it is not deterministic, to reproduce
this, you can set server `accounting.oom.critical.heap.usage.ratio` to be
higher in `OfflineClusterMemBasedServerQueryKillingTest` and try running
`testSelectionOnlyOOM` .
```
{"message":"QueryCancellationError:\norg.apache.pinot.spi.exception.QueryCancelledException:
Cancelled while building data table java.lang.RuntimeException: Query
Broker_172.25.200.137_18099_752082720000000006_O got killed because using
2353711296 bytes of memory on SERVER: Server_localhost_8098, exceeding the
quota\n\tat
org.apache.pinot.core.query.scheduler.QueryScheduler.serializeResponse(QueryScheduler.java:227)\n\tat
org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:156)\n\tat
org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:124)\n\tat
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n...\nCaused
by: org.apache.pinot.spi.exception.EarlyTerminationException: Interrupted while
merging records\n\tat
org.apache.pinot.spi.trace.Tracing$ThreadAccountantOps.sampleAndCheckInterruption(Tracing.java:288)\n\tat
org.apache.pinot.spi.trace.Tracing$ThreadAccountantOps.s
ampleAndCheckInterruptionPeriodically(Tracing.java:304)\n\tat
org.apache.pinot.core.query.selection.SelectionOperatorUtils.getDataTableFromRows(SelectionOperatorUtils.java:388)\n\tat
org.apache.pinot.core.operator.blocks.results.SelectionResultsBlock.getDataTable(SelectionResultsBlock.java:84)","errorCode":503}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
