Github user Dooyoung-Hwang commented on the issue:
https://github.com/apache/spark/pull/22219
Yes, I verified results of a variety of queries, and memory & performance.
This patch passed all our query test. And there was no performance
degradation in our test cases.
And below is result of memory test.
I checked **memory utilization of Old Gen in JVM Heap** when executes query
of **2,481,284 rows**. (I Executed "jstat -gc thriftserver-pid" and checked OU
field.)
After patch : 283910.0KB -> 316108.3KB => 31.44MB increases
Before patch : 279425.6KB -> 1511834.2KB => 1203.52MB increases
Memory improvement is very large, because the size of compressed result
buffer surprisingly smaller than I expected. **Decompressed InternalRows are
collected immediately after sending them while Young GC is done**, so the usage
of Old Gen Heap is much smaller than before.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
