Jakub Nowacki created SPARK-21752:
-------------------------------------
Summary: Config spark.jars.packages is ignored in SparkSession
config
Key: SPARK-21752
URL: https://issues.apache.org/jira/browse/SPARK-21752
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Reporter: Jakub Nowacki
If I put a config key {{spark.jars.packages}} using {{SparkSession}} builder as
follows:
{code}
spark = pyspark.sql.SparkSession.builder\
.appName('test-mongo')\
.master('local[*]')\
.config("spark.jars.packages",
"org.mongodb.spark:mongo-spark-connector_2.11:2.2.0"))\
.config("spark.mongodb.input.uri", "mongodb://mongo/test.coll") \
.config("spark.mongodb.output.uri", "mongodb://mongo/test.coll") \
.getOrCreate()
{code}
the SparkSession gets created but there are no package download logs printed,
and if I use the loaded classes, Mongo connector in this case, but it's the
same for other packages, I get {{java.lang.ClassNotFoundException}} for the
missing classes.
If I use the config file {{conf/spark-defaults.comf}}, command line option
{{--packages}}, e.g.:
{code}
import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages
org.mongodb.spark:mongo-spark-connector_2.11:2.2.0 pyspark-shell'
{code}
it works fine. Interestingly, using {{SparkConf}} object works fine as well,
e.g.:
{code}
conf = pyspark.SparkConf()
conf.set("spark.jars.packages",
"org.mongodb.spark:mongo-spark-connector_2.11:2.2.0")
conf.set("spark.mongodb.input.uri", "mongodb://mongo/test.coll")
conf.set("spark.mongodb.output.uri", "mongodb://mongo/test.coll")
spark = pyspark.sql.SparkSession.builder\
.appName('test-mongo')\
.master('local[*]')\
.config(conf=conf)\
.getOrCreate()
{code}
The above is in Python but I've seen the behavior in other languages, though, I
didn't check R.
I also have seen it in older Spark versions.
It seems that this is the only config key that doesn't work for me via the
{{SparkSession}} builder config.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
