Quanlong Huang created IMPALA-12949:
---------------------------------------
Summary: Need profile counters of BufferPool for
UndoMoveEvictedToPinned()
Key: IMPALA-12949
URL: https://issues.apache.org/jira/browse/IMPALA-12949
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Quanlong Huang
Assignee: Quanlong Huang
When comparing profiles of two different runs of the same query, I got confused
by the BufferPool counters:
{noformat}
Buffer pool | Buffer pool
- AllocTime: 23.74s (23741060380) | - AllocTime: 7.47s
(7466380924)
- CompressionTime: 0ns (0) | - CompressionTime: 0ns
(0)
- CumulativeAllocationBytes: 32.9GiB (35320410931) | -
CumulativeAllocationBytes: 28.6GiB (30664333721)
- CumulativeAllocations: 473,584 (473584) | -
CumulativeAllocations: 402,590 (402590)
- EncryptionTime: 0ns (0) | - EncryptionTime: 0ns
(0)
- InactiveTotalTime: 0ns (0) | - InactiveTotalTime:
0ns (0)
- PeakReservation: 535.2MiB (561158553) | - PeakReservation:
484.1MiB (507615641)
- PeakUnpinnedBytes: 10.0GiB (10760834252) | - PeakUnpinnedBytes:
10.0GiB (10760939110)
- PeakUsedReservation: 534.8MiB (560798105) | - PeakUsedReservation:
483.9MiB (507419033)
- ReadIoBytes: 9.4GiB (10051446374) | - ReadIoBytes: 5.0GiB
(5354822041)
- ReadIoOps: 153,372 (153372) | - ReadIoOps: 81,708
(81708)
- ReadIoWaitTime: 43.1m (2586172694497) | - ReadIoWaitTime: 2.1m
(128591149546)
- SystemAllocTime: 21.71s (21707880137) | - SystemAllocTime:
4.75s (4754840861)
- TotalTime: 0ns (0) | - TotalTime: 0ns (0)
- WriteIoBytes: 17.4GiB (18726643302) | - WriteIoBytes:
17.3GiB (18590570905)
- WriteIoOps: 285,745 (285745) | - WriteIoOps: 283,669
(283669)
- WriteIoWaitTime: 36.78s (36777011711) | - WriteIoWaitTime:
1.1m (65635211353){noformat}
WriteIoBytes and WriteIoOps are close. But ReadIoBytes and ReadIoOps on the
left are 2x of the right. It seems the operator is spilling the same amount of
data but different in reading it back.
Read through the codes, the ReadIo* counters are updated in
BufferPool::Client::StartMoveEvictedToPinned() which starts an async read [1].
It can be cancelled by BufferPool::Client::UndoMoveEvictedToPinned(). However,
there are no counters showing how many times UndoMoveEvictedToPinned() is
invoked. We should add such a counter to explain the difference in ReadIoBytes
and ReadIoOps.
[1]
https://github.com/apache/impala/blob/c8d56425f87f2bc6da5fd39ed905caeabc9aaad0/be/src/runtime/bufferpool/buffer-pool.cc#L560-L561
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
