[ https://issues.apache.org/jira/browse/IMPALA-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819702#comment-16819702 ]
ASF subversion and git services commented on IMPALA-8375: --------------------------------------------------------- Commit a80ebb7a729c1ffd25db262527d4261e01763646 in impala's branch refs/heads/master from Abhishek Rawat [ https://gitbox.apache.org/repos/asf?p=impala.git;h=a80ebb7 ] IMPALA-8375: Add metrics for spill disk usage Added two new metrics tmp-file-mgr.scratch-space-bytes-used-high-water-mark & tmp-file-mgr.scratch-space-bytes-used for tracking HWM and current value for spilled bytes, respectively. A new class AtomicHighWaterMarkGauge was added to keep track of the HWM value. The new class also encapsulates a metric object which keeps track of the current value for the spilled bytes. The current value is incremented every time a new range is allocated from a temporary file. The current value for spilled bytes is decremented when a temporary file is closed. The new metrics are not updated when ranges are recycled from a file. We can add a new metric in future for keeping track of actual spilled bytes. The HWM value is updated whenever the current value is greater than the HWM value. Testing: - Added new unit tests to the metrics-test test case. - E2E testing for both the metrics by running concurrent spilling queries and ensuring that both the current value metric and the HWM metric were behaving as expected. Ran concurrent queries and monitored the metrics on the impala daemon's metric page. Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Reviewed-on: http://gerrit.cloudera.org:8080/12956 Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > Add metrics for spill disk usage > -------------------------------- > > Key: IMPALA-8375 > URL: https://issues.apache.org/jira/browse/IMPALA-8375 > Project: IMPALA > Issue Type: Sub-task > Components: Backend > Affects Versions: Impala 3.2.0 > Reporter: Tim Armstrong > Assignee: Abhishek Rawat > Priority: Major > > * High water mark of bytes allocated > * Maybe - some metric of utilisation (what % of bytes allocated are actually > being used) > * Maybe - total bytes allocated over the lifetime of the process > * Maybe - statistics about the sizes of blocks allocated -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org