[
https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824604#comment-16824604
]
Dharmesh Kakadia commented on SPARK-27287:
------------------------------------------
Understood. I do not know how to "use sparkSession" when reading ML model,
because PCAModel.load() for example has know way to specify/use a given
sparksession.
if you think it is relatively easy to fix (copying the configs to context), I
can send a patch. I would need help on that though.
> PCAModel.load() does not honor spark configs
> --------------------------------------------
>
> Key: SPARK-27287
> URL: https://issues.apache.org/jira/browse/SPARK-27287
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 2.4.0
> Reporter: Dharmesh Kakadia
> Priority: Major
>
> PCAModel.load() does not seem to be using the configurations set on the
> current spark session.
> Repro:
>
> The following will fail to read the data because the storage account
> credentials config used/propagated.
> conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")
> spark =
> SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()
> model = PCAModel.load('wasb://[email protected]/model')
>
> The following however works:
> conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")
> spark =
> SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()
> blah =
> spark.read.json('wasb://[email protected]/somethingelse/')
> blah.show()
> model = PCAModel.load('wasb://[email protected]/model')
>
> It looks like spark.read...() does force the use of the config once and then
> PCAModel.load() will work correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
