Dharmesh Kakadia created SPARK-27287:
----------------------------------------
Summary: PCAModel.load() does not honor spark configs
Key: SPARK-27287
URL: https://issues.apache.org/jira/browse/SPARK-27287
Project: Spark
Issue Type: Bug
Components: ML
Affects Versions: 2.4.0
Reporter: Dharmesh Kakadia
PCAModel.load() does not seem to be using the configurations set on the current
spark session.
Repro:
The following will fail to read the data because the storage account
credentials config used/propagated.
conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")
spark =
SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()
model = PCAModel.load('wasb://[email protected]/model')
The following however works:
conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")
spark =
SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()
blah = spark.read.json('wasb://[email protected]/somethingelse/')
blah.show()
model = PCAModel.load('wasb://[email protected]/model')
It looks like spark.read...() does force the use of the config once and then
PCAModel.load() will work correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
