Russell Spitzer created SPARK-25003:
---------------------------------------
Summary: Pyspark Does not use Spark Sql Extensions
Key: SPARK-25003
URL: https://issues.apache.org/jira/browse/SPARK-25003
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.3.1, 2.2.2
Reporter: Russell Spitzer
When creating a SparkSession here
[https://github.com/apache/spark/blob/v2.2.2/python/pyspark/sql/session.py#L216]
{code:python}
if jsparkSession is None:
jsparkSession = self._jvm.SparkSession(self._jsc.sc())
self._jsparkSession = jsparkSession
{code}
I believe it ends up calling the constructor here
https://github.com/apache/spark/blob/v2.2.2/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala#L85-L87
{code:scala}
private[sql] def this(sc: SparkContext) {
this(sc, None, None, new SparkSessionExtensions)
}
{code}
Which creates a new SparkSessionsExtensions object and does not pick up new
extensions that could have been set in the config like the companion
getOrCreate does.
https://github.com/apache/spark/blob/v2.2.2/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala#L928-L944
{code:scala}
//in getOrCreate
// Initialize extensions if the user has defined a configurator class.
val extensionConfOption =
sparkContext.conf.get(StaticSQLConf.SPARK_SESSION_EXTENSIONS)
if (extensionConfOption.isDefined) {
val extensionConfClassName = extensionConfOption.get
try {
val extensionConfClass = Utils.classForName(extensionConfClassName)
val extensionConf = extensionConfClass.newInstance()
.asInstanceOf[SparkSessionExtensions => Unit]
extensionConf(extensions)
} catch {
// Ignore the error if we cannot find the class or when the class
has the wrong type.
case e @ (_: ClassCastException |
_: ClassNotFoundException |
_: NoClassDefFoundError) =>
logWarning(s"Cannot use $extensionConfClassName to configure
session extensions.", e)
}
}
{code}
I think a quick fix would be to use the getOrCreate method from the companion
object instead of calling the constructor from the SparkContext. Or we could
fix this by ensuring that all constructors attempt to pick up custom extensions
if they are set.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
