[
https://issues.apache.org/jira/browse/SPARK-48423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109909#comment-18109909
]
Nguyễn Hoàng Gia Bảo commented on SPARK-48423:
----------------------------------------------
*1. Reproduction*
I reproduced this issue locally using PySpark. Even without Azure dependencies
installed (which throws a {{ClassNotFoundException}} instead of
{{InvalidConfigurationValueException}}), the stack trace perfectly reveals the
execution path and configuration loss.
{code:python}
from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer
from pyspark.ml import Pipeline
spark =
SparkSession.builder.appName("SPARK-48423-Reproduce").master("local[2]").getOrCreate()
df = spark.createDataFrame([(0, "a b c d e spark")], ["id", "text"])
pipelineModel = Pipeline(stages=[StringIndexer(inputCol="text",
outputCol="IND_text")]).fit(df)
path = "abfs://[email protected]/test_model"
# Passing credentials via .option()
pipelineModel.write() \
.option("fs.azure.account.key.fakeaccount.dfs.core.windows.net",
"dummy_secret_key") \
.save(path)
{code}
*2. Root Cause*
The issue originates in {{org.apache.spark.ml.util.MLWriter.save}} (inside
{{ReadWrite.scala}}).
When {{.save(path)}} is called, it triggers filesystem operations (e.g.,
{{FileSystemOverwrite.handleOverwrite}} or creating output streams) which
eventually call {{path.getFileSystem(sc.hadoopConfiguration)}}.
While {{MLWriter}} correctly stores the user-provided options in its internal
{{optionMap}}, it fails to propagate these options to the underlying Hadoop
configuration before instantiating the {{FileSystem}}. As a result, the global
{{sc.hadoopConfiguration}} is used directly, and all Azure credentials passed
via {{.option()}} are completely ignored by the Hadoop I/O layer.
*3. Proposed Fix*
To fix this, {{MLWriter.save()}} (and similarly {{MLReader.load()}}) should
construct a localized Hadoop {{Configuration}} object that merges the global
{{sc.hadoopConfiguration}} with the key-value pairs stored in {{optionMap}}.
This enriched configuration should then be passed to all Hadoop
{{Path.getFileSystem()}} or I/O calls within the ML persistence lifecycle,
ensuring consistency with how {{DataFrameWriter}} handles {{.options()}}.
> Unable to write MLPipeline to blob storage using .option attribute
> ------------------------------------------------------------------
>
> Key: SPARK-48423
> URL: https://issues.apache.org/jira/browse/SPARK-48423
> Project: Spark
> Issue Type: Bug
> Components: ML, MLlib, Spark Core
> Affects Versions: 3.4.3
> Reporter: Chhavi Bansal
> Priority: Critical
>
> I am trying to write mllib pipeline with a series of stages set in it to
> azure blob storage giving relevant write parameters, but it still complains
> of `fs.azure.account.key` not being found in the configuration.
> Sharing the code.
> {code:java}
> val spark =
> SparkSession.builder().appName("main").master("local[4]").getOrCreate()
> import spark.implicits._
> val df = spark.createDataFrame(Seq(
> (0L, "a b c d e spark"),
> (1L, "b d")
> )).toDF("id", "text")
> val si = new StringIndexer().setInputCol("text").setOutputCol("IND_text")
> val pipelinee = new Pipeline().setStages(Array(si))
> val pipelineModel = pipelinee.fit(df)
> val path = BLOB_STORAGE_PATH
> pipelineModel.write
> .option("spark.hadoop.fs.azure.account.key.<account_name>.dfs.core.windows.net",
> "__").option("fs.azure.account.key.<account_name>.dfs.core.windows.net",
> "__").option("fs.azure.account.oauth2.client.endpoint.<account_name>.dfs.core.windows.net",
>
> "__").option("fs.azure.account.oauth2.client.id.<account_name>.dfs.core.windows.net",
>
> "__").option("fs.azure.account.auth.type.<account_name>.dfs.core.windows.net","__").option("fs.azure.account.oauth2.client.secret.<account_name>.dfs.core.windows.net",
>
> "__").option("fs.azure.account.oauth.provider.type.<account_name>.dfs.core.windows.net",
> "__")
> .save(path){code}
>
> The error that i get is
> {code:java}
> Failure to initialize configuration
> Caused by: InvalidConfigurationValueException: Invalid configuration value
> detected for fs.azure.account.key
> at
> org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:51)
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:548)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1449){code}
> This shows that even though the key,value of
> {code:java}
> spark.hadoop.fs.azure.account.key.<account_name>.dfs.core.windows.net {code}
> is being sent via option param, but is not being set internally.
>
> while this works only if i explicitly set the values in the
> {code:java}
> spark.conf.set(key,value) {code}
> which might be problematic for a multi-tenant solution, which can be using
> the same spark context.
> one other observation is
> {code:java}
> df.write.option(key1,value1).option(key2,value2).save(path) {code}
> fails with same key error while,
> {code:java}
> map = Map(key1->value1, key2->value2)
> df.write.options(map).save(path) {code}
> works..
>
> Help required on: Similar to how dataframes `options`
> {code:java}
> df.write.options(Map<key,value>) {code}
> helps to set the configuration, the *.option(key1, value1)* should also work
> to write to azure blob storage.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
