[
https://issues.apache.org/jira/browse/IMPALA-8537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Rorke updated IMPALA-8537:
--------------------------------
Description:
I'm running a workload that does a lot of spilling and noticed the value
reported for tmp-file-mgr.scratch-space-bytes-used is negative on all nodes.
Some details of the workload and cluster configuration:
* Generating a 10 TB TPC-DS partitioned parquet data set (very large sort).
* 30 impalads, each with 48 GB RAM and 14 scratch directories (each on a
separate drive)
* Rough estimate (based on query metrics) of total cumulative aggregate memory
spilled across the cluster since restart is 6.5 TB.
Snapshot of the bad metrics attached.
was:
I'm running a workload that does a lot of spilling and noticed the value
reported for tmp-file-mgr.scratch-space-bytes-used is negative on all nodes.
Some details of the workload and cluster configuration:
* Generating a 10 TB TPC-DS partitioned parquet data set (very large sort).
* 30 impalads, each with 48 GB RAM and 14 scratch directories (each on a
separate drive)
* Rough estimate (based on query metrics) of total aggregate memory spilled
across the cluster since restart is 6.5 TB.
Snapshot of the bad metrics attached.
> Negative values reported for tmp-file-mgr.scratch-space-bytes-used under
> heavy spilling load
> --------------------------------------------------------------------------------------------
>
> Key: IMPALA-8537
> URL: https://issues.apache.org/jira/browse/IMPALA-8537
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.2.0
> Reporter: David Rorke
> Assignee: Abhishek Rawat
> Priority: Major
> Attachments: bad_spill_metrics.json
>
>
> I'm running a workload that does a lot of spilling and noticed the value
> reported for tmp-file-mgr.scratch-space-bytes-used is negative on all nodes.
> Some details of the workload and cluster configuration:
> * Generating a 10 TB TPC-DS partitioned parquet data set (very large sort).
> * 30 impalads, each with 48 GB RAM and 14 scratch directories (each on a
> separate drive)
> * Rough estimate (based on query metrics) of total cumulative aggregate
> memory spilled across the cluster since restart is 6.5 TB.
> Snapshot of the bad metrics attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
