GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/13245
[SPARK-15466][SQL] Make `SparkSession` as the entry point to programming
with RDD too
## What changes were proposed in this pull request?
`SparkSession` greatly reduces the number of concepts which Spark users
must know. Currently, `SparkSession` is defined as the entry point to
programming Spark with the Dataset and DataFrame API. And, we can easily get
`RDD` by calling `Dataset.rdd` or `DataFrame.rdd`, too.
However, many usages (including examples) are observed to extract
`SparkSession.sparkContext` and keep it as own variable to call `parallelize`.
If `SparkSession` supports RDD seamlessly too, it would be great for
usability. We can do this by simply adding `parallelize` API.
**Example**
```scala
object SparkPi {
def main(args: Array[String]) {
val spark = SparkSession
.builder
.appName("Spark Pi")
.getOrCreate()
- val sc = spark.sparkContext
val slices = if (args.length > 0) args(0).toInt else 2
val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid
overflow
- val count = sc.parallelize(1 until n, slices).map { i =>
+ val count = spark.parallelize(1 until n, slices).map { i =>
val count = spark.parallelize(1 until n, slices).map { i =>
val x = random * 2 - 1
val y = random * 2 - 1
if (x*x + y*y < 1) 1 else 0
}.reduce(_ + _)
println("Pi is roughly " + 4.0 * count / n)
spark.stop()
}
}
```
```python
spark = SparkSession\
.builder\
.appName("PythonPi")\
.getOrCreate()
- sc = spark._sc
-
partitions = int(sys.argv[1]) if len(sys.argv) > 1 else 2
n = 100000 * partitions
def f(_):
x = random() * 2 - 1
y = random() * 2 - 1
return 1 if x ** 2 + y ** 2 < 1 else 0
-count = sc.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
count = spark.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
print("Pi is roughly %f" % (4.0 * count / n))
spark.stop()
```
## How was this patch tested?
Pass the Jenkins test (with new python test) and also manual.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-15466
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13245.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13245
----
commit 810f08a666c5d14a2178e329b7c1727603be485e
Author: Dongjoon Hyun <[email protected]>
Date: 2016-05-21T21:36:04Z
[SPARK-15466][SQL] Make `SparkSession` as the entry point to programming
with RDD too
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
