Github user davies commented on the issue:
https://github.com/apache/spark/pull/11956
@kiszk The current implementation use ByteBuffer and smart compression
algorithms, it too slow to build the in-memory cache, make it useless. So we'd
like to improve the performance of building phase also. This PR is built for
current representation, which may be throwed away in future, so I'd not rush to
merge this PR, or spent to much time to review the details.
PR #13899 could be in the right direction, but need to double-check that by
more benchmarks. In order to have better memory efficiency, we could use
MEMORY_AND_SER storage level and compress the underlying array with LZ4 when
serializing the ColumnVector.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
