GitHub user ericl opened a pull request:
https://github.com/apache/spark/pull/20971
[SPARK-23809][SQL][backport] Active SparkSession should be set by
getOrCreate
This backports https://github.com/apache/spark/pull/20927 to branch-2.3
## What changes were proposed in this pull request?
Currently, the active spark session is set inconsistently (e.g., in
createDataFrame, prior to query execution). Many places in spark also
incorrectly query active session when they should be calling
activeSession.getOrElse(defaultSession) and so might get None even if a Spark
session exists.
The semantics here can be cleaned up if we also set the active session when
the default session is set.
Related: https://github.com/apache/spark/pull/20926/files
## How was this patch tested?
Unit test, existing test. Note that if
https://github.com/apache/spark/pull/20926 merges first we should also update
the tests there.
Author: Eric Liang <[email protected]>
Closes #20927 from ericl/active-session-cleanup.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ericl/spark backport-23809
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20971.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20971
----
commit f2303dcef61660dabfd08be5568b7da10cf1b117
Author: Eric Liang <ekl@...>
Date: 2018-04-04T00:09:12Z
[SPARK-23809][SQL] Active SparkSession should be set by getOrCreate
## What changes were proposed in this pull request?
Currently, the active spark session is set inconsistently (e.g., in
createDataFrame, prior to query execution). Many places in spark also
incorrectly query active session when they should be calling
activeSession.getOrElse(defaultSession) and so might get None even if a Spark
session exists.
The semantics here can be cleaned up if we also set the active session when
the default session is set.
Related: https://github.com/apache/spark/pull/20926/files
## How was this patch tested?
Unit test, existing test. Note that if
https://github.com/apache/spark/pull/20926 merges first we should also update
the tests there.
Author: Eric Liang <[email protected]>
Closes #20927 from ericl/active-session-cleanup.
commit e429af1e9a5a2f8ed3e90ee215d561c05aeb33b3
Author: Eric Liang <ekl@...>
Date: 2018-04-04T00:30:50Z
Tue Apr 3 17:30:50 PDT 2018
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
