Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17436
after some more thoughts, I think we should not rely on
`spark.memory.offHeap.enabled`. The reason is that: 1) `OffHeapColumnVector`
may have performance regression, users may want off-heap memory manager but
still use `OnHeapColumnVector`. 2) `OffHeapColumnVector` doesn't allocate
memory from memory manager, so the off-heap size has no effect to
`OffHeapColumnVector`.
I think we should introduce a new config
`spark.sql.columnVector.offheap.enable` for this new behavior.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
