GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/2513
[SPARK-3645][SQL] Makes table caching eager by default and adds syntax for
lazy caching
Although lazy caching for in-memory table seems consistent with the
`RDD.cache()` API, it's relatively confusing for users who mainly work with SQL
and not familiar with Spark internals. The `CACHE TABLE t; SELECT COUNT(*) FROM
t;` pattern is also commonly seen just to ensure predictable performance.
This PR makes both the `CACHE TABLE t [AS SELECT ...]` statement and the
`SQLContext.cacheTable()` API eager by default, and adds a new `LAZY CACHE
TABLE t [AS SELECT ...]` syntax to provide lazy in-memory table caching.
Also, took the chance to make some refactoring: `CacheCommand` and
`CacheTableAsSelectCommand` are now merged and renamed to `CacheTableCommand`
since the former is strictly a special case of the latter. A new
`UncacheTableCommand` is added for the `UNCACHE TABLE t` statement.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark eager-caching
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2513.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2513
----
commit 0fc0c15645283b606d4056998c74d7eae643c17a
Author: Cheng Lian <[email protected]>
Date: 2014-09-23T05:28:25Z
Made cacheTable() and CACHE TABLE statements eager
commit b72e24ebf950f338dc1829eec4fd3ed81ed1b62e
Author: Cheng Lian <[email protected]>
Date: 2014-09-24T00:42:32Z
Adds LAZY CACHE TABLE syntax
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
