GitHub user feynmanliang opened a pull request:
https://github.com/apache/spark/pull/8524
[SQL] Adds private implicit tungstenCache to DataFrame
Adds `tungstenCache` to `DataFrame`, which caches the `DataFrame`'s rows
using contiguous blocks of off-heap memory.
CC @rxin @marmbrus
Closes #8523 (changes there are included, can also rebase if easier)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/feynmanliang/spark tungsten-cache
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8524.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8524
----
commit 3c42e21745997c3aa6d2ea84ea920b7421b34c0c
Author: Feynman Liang <[email protected]>
Date: 2015-08-29T23:28:57Z
Adds handling for java.lang.String on getUTF8String
commit b907a71ba20ba7283b4e7962f3d6db7fa9b793da
Author: Feynman Liang <[email protected]>
Date: 2015-08-29T23:31:18Z
Fixes whitespace and wording
commit 0c6d10d638ff6848da468518ce7d9f9e358be4af
Author: Feynman Liang <[email protected]>
Date: 2015-08-29T23:51:56Z
Adds test causing NullPointerException
commit aa84a8e21040b60a922749b7c736937c52b1eebe
Author: Feynman Liang <[email protected]>
Date: 2015-08-29T23:57:50Z
Fixes getUTF8String to support off-heap backed memory
commit 26aa8d31a65f2adf8eee71a3526deac6815e82b0
Author: Feynman Liang <[email protected]>
Date: 2015-08-29T01:07:39Z
Adds project boilerplate
commit 2a5868b26c8c7b9ceb3ef912f71ca0e6d285d86f
Author: Feynman Liang <[email protected]>
Date: 2015-08-29T22:44:45Z
Gets to working example, fixes bug in UnsafeRow.getUTF8String
commit f16299dd1949a3bf71d5d9b5023f800de0fe5cfe
Author: Feynman Liang <[email protected]>
Date: 2015-08-30T00:15:57Z
Makes tests work and cleans up code
commit 50d65cf91ec1eb1c407c864008066c3d7d2a283d
Author: Feynman Liang <[email protected]>
Date: 2015-08-30T00:49:19Z
Adds tungsten caching
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
