Repository: spark
Updated Branches:
refs/heads/master 40a949aae -> de9c85cca
[SPARK-15270] [SQL] Use SparkSession Builder to build a session with HiveSupport
## What changes were proposed in this pull request?
Before:
Creating a hiveContext was failing
```python
from pyspark.sql import HiveContext
hc = HiveContext(sc)
```
with
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "spark-2.0/python/pyspark/sql/context.py", line 458, in __init__
sparkSession = SparkSession.withHiveSupport(sparkContext)
File "spark-2.0/python/pyspark/sql/session.py", line 192, in withHiveSupport
jsparkSession =
sparkContext._jvm.SparkSession.withHiveSupport(sparkContext._jsc.sc())
File "spark-2.0/python/lib/py4j-0.9.2-src.zip/py4j/java_gateway.py", line
1048, in __getattr__
py4j.protocol.Py4JError: org.apache.spark.sql.SparkSession.withHiveSupport does
not exist in the JVM
```
Now:
```python
>>> from pyspark.sql import HiveContext
>>> hc = HiveContext(sc)
>>> hc.range(0, 100)
DataFrame[id: bigint]
>>> hc.range(0, 100).count()
100
```
## How was this patch tested?
Existing Tests, tested manually in python shell
Author: Sandeep Singh <[email protected]>
Closes #13056 from techaddict/SPARK-15270.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/de9c85cc
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/de9c85cc
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/de9c85cc
Branch: refs/heads/master
Commit: de9c85ccaacd12de9837eb88eae0a7e7ededd679
Parents: 40a949a
Author: Sandeep Singh <[email protected]>
Authored: Wed May 11 14:15:18 2016 -0700
Committer: Davies Liu <[email protected]>
Committed: Wed May 11 14:15:18 2016 -0700
----------------------------------------------------------------------
python/pyspark/sql/context.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/de9c85cc/python/pyspark/sql/context.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/context.py b/python/pyspark/sql/context.py
index 78ab2e8..02e742c 100644
--- a/python/pyspark/sql/context.py
+++ b/python/pyspark/sql/context.py
@@ -455,7 +455,7 @@ class HiveContext(SQLContext):
def __init__(self, sparkContext, jhiveContext=None):
if jhiveContext is None:
- sparkSession = SparkSession.withHiveSupport(sparkContext)
+ sparkSession =
SparkSession.builder.enableHiveSupport().getOrCreate()
else:
sparkSession = SparkSession(sparkContext,
jhiveContext.sparkSession())
SQLContext.__init__(self, sparkContext, sparkSession, jhiveContext)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
