[
https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807780#comment-16807780
]
Marco Gaido commented on SPARK-27287:
-------------------------------------
I think the problem here is that the configuration are copied from the
SparkSession to the SparkContext when the SparkSession is used, but they are
not if the SparkContext is used (as it is done when loading a model in ML). I
think the solutions here may be 2: fix the way contexts are handling properties
(preferred); use the sparkSession when reading ML models (easier workaround).
> PCAModel.load() does not honor spark configs
> --------------------------------------------
>
> Key: SPARK-27287
> URL: https://issues.apache.org/jira/browse/SPARK-27287
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 2.4.0
> Reporter: Dharmesh Kakadia
> Priority: Major
>
> PCAModel.load() does not seem to be using the configurations set on the
> current spark session.
> Repro:
>
> The following will fail to read the data because the storage account
> credentials config used/propagated.
> conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")
> spark =
> SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()
> model = PCAModel.load('wasb://[email protected]/model')
>
> The following however works:
> conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")
> spark =
> SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()
> blah =
> spark.read.json('wasb://[email protected]/somethingelse/')
> blah.show()
> model = PCAModel.load('wasb://[email protected]/model')
>
> It looks like spark.read...() does force the use of the config once and then
> PCAModel.load() will work correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
