[GitHub] [spark] cloud-fan commented on pull request #28780: [SPARK-31952][SQL]Fix incorrect memory spill metric when doing Aggregate

2020-06-23 Thread GitBox
cloud-fan commented on pull request #28780: URL: https://github.com/apache/spark/pull/28780#issuecomment-647945904 shall we set `sorter.totalSpillBytes`, then we can update the metrics correctly in `sort.spill`. This is an

[GitHub] [spark] cloud-fan commented on pull request #28780: [SPARK-31952][SQL]Fix incorrect memory spill metric when doing Aggregate

2020-06-21 Thread GitBox
cloud-fan commented on pull request #28780: URL: https://github.com/apache/spark/pull/28780#issuecomment-647288933 > number of in-memory bytes spilled what's the difference between it and `spill (disk)`? IIUC `spill` means we dump data from memory to disk.

[GitHub] [spark] cloud-fan commented on pull request #28780: [SPARK-31952][SQL]Fix incorrect memory spill metric when doing Aggregate

2020-06-16 Thread GitBox
cloud-fan commented on pull request #28780: URL: https://github.com/apache/spark/pull/28780#issuecomment-644697788 I'm not familiar with this part. what does "spill (memory)" mean? This is an automated message from the