[
https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130282#comment-16130282
]
Jakub Nowacki commented on SPARK-21752:
---------------------------------------
[~skonto] Well, I'm not sure where you're failing here. If you want to get
PySpark installed with a vanilla Python distribution you can do {{pip install
pyspark}} or {{conda install -c conda-forge pyspark}}. Other that that, the
above scripts are complete, bar the {{import pyspark}} as I mentioned before.
Below I give a sightly more complete example with the env variable:
{code}
import os
import pyspark
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages
org.mongodb.spark:mongo-spark-connector_2.11:2.2.0 pyspark-shell'
spark = pyspark.sql.SparkSession.builder\
.appName('test-mongo')\
.master('local[*]')\
.config("spark.mongodb.input.uri", "mongodb://mongo/test.coll") \
.config("spark.mongodb.output.uri", "mongodb://mongo/test.coll") \
.getOrCreate()
l = [("Bilbo Baggins", 50),
("Gandalf", 1000),
("Thorin", 195),
("Balin", 178),
("Kili", 77),
("Dwalin", 169),
("Oin", 167),
("Gloin", 158),
("Fili", 82),
("Bombur", None)]
people = spark.createDataFrame(l, ["name", "age"])
people.write \
.format("com.mongodb.spark.sql.DefaultSource") \
.mode("append") \
.save()
spark.read \
.format("com.mongodb.spark.sql.DefaultSource") \
.load() \
.show()
{code}
and with the {{SparkConfig}} approach:
{code}
import pyspark
conf = pyspark.SparkConf()
conf.set("spark.jars.packages",
"org.mongodb.spark:mongo-spark-connector_2.11:2.2.0")
conf.set("spark.mongodb.input.uri", "mongodb://mongo/test.coll")
conf.set("spark.mongodb.output.uri", "mongodb://mongo/test.coll")
spark = pyspark.sql.SparkSession.builder\
.appName('test-mongo')\
.master('local[*]')\
.config(conf=conf)\
.getOrCreate()
l = [("Bilbo Baggins", 50),
("Gandalf", 1000),
("Thorin", 195),
("Balin", 178),
("Kili", 77),
("Dwalin", 169),
("Oin", 167),
("Gloin", 158),
("Fili", 82),
("Bombur", None)]
people = spark.createDataFrame(l, ["name", "age"])
people.write \
.format("com.mongodb.spark.sql.DefaultSource") \
.mode("append") \
.save()
spark.read \
.format("com.mongodb.spark.sql.DefaultSource") \
.load() \
.show()
{code}
and with the plain {{SparkSession}} config:
{code}
import pyspark
spark = pyspark.sql.SparkSession.builder\
.appName('test-mongo')\
.master('local[*]')\
.config("spark.jars.packages",
"org.mongodb.spark:mongo-spark-connector_2.11:2.2.0")\
.config("spark.mongodb.input.uri", "mongodb://mongo/test.coll") \
.config("spark.mongodb.output.uri", "mongodb://mongo/test.coll") \
.getOrCreate()
l = [("Bilbo Baggins", 50),
("Gandalf", 1000),
("Thorin", 195),
("Balin", 178),
("Kili", 77),
("Dwalin", 169),
("Oin", 167),
("Gloin", 158),
("Fili", 82),
("Bombur", None)]
people = spark.createDataFrame(l, ["name", "age"])
people.write \
.format("com.mongodb.spark.sql.DefaultSource") \
.mode("append") \
.save()
spark.read \
.format("com.mongodb.spark.sql.DefaultSource") \
.load() \
.show()
{code}
In my case the first two work as expected, and the last one fails with
{{ClassNotFoundException}}.
> Config spark.jars.packages is ignored in SparkSession config
> ------------------------------------------------------------
>
> Key: SPARK-21752
> URL: https://issues.apache.org/jira/browse/SPARK-21752
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Jakub Nowacki
>
> If I put a config key {{spark.jars.packages}} using {{SparkSession}} builder
> as follows:
> {code}
> spark = pyspark.sql.SparkSession.builder\
> .appName('test-mongo')\
> .master('local[*]')\
> .config("spark.jars.packages",
> "org.mongodb.spark:mongo-spark-connector_2.11:2.2.0")\
> .config("spark.mongodb.input.uri", "mongodb://mongo/test.coll") \
> .config("spark.mongodb.output.uri", "mongodb://mongo/test.coll") \
> .getOrCreate()
> {code}
> the SparkSession gets created but there are no package download logs printed,
> and if I use the loaded classes, Mongo connector in this case, but it's the
> same for other packages, I get {{java.lang.ClassNotFoundException}} for the
> missing classes.
> If I use the config file {{conf/spark-defaults.comf}}, command line option
> {{--packages}}, e.g.:
> {code}
> import os
> os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages
> org.mongodb.spark:mongo-spark-connector_2.11:2.2.0 pyspark-shell'
> {code}
> it works fine. Interestingly, using {{SparkConf}} object works fine as well,
> e.g.:
> {code}
> conf = pyspark.SparkConf()
> conf.set("spark.jars.packages",
> "org.mongodb.spark:mongo-spark-connector_2.11:2.2.0")
> conf.set("spark.mongodb.input.uri", "mongodb://mongo/test.coll")
> conf.set("spark.mongodb.output.uri", "mongodb://mongo/test.coll")
> spark = pyspark.sql.SparkSession.builder\
> .appName('test-mongo')\
> .master('local[*]')\
> .config(conf=conf)\
> .getOrCreate()
> {code}
> The above is in Python but I've seen the behavior in other languages, though,
> I didn't check R.
> I also have seen it in older Spark versions.
> It seems that this is the only config key that doesn't work for me via the
> {{SparkSession}} builder config.
> Note that this is related to creating new {{SparkSession}} as getting new
> packages into existing {{SparkSession}} doesn't indeed make sense. Thus this
> will only work with bare Python, Scala or Java, and not on {{pyspark}} or
> {{spark-shell}} as they create the session automatically; it this case one
> would need to use {{--packages}} option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
