beliefer opened a new pull request, #52693:
URL: https://github.com/apache/spark/pull/52693
### What changes were proposed in this pull request?
This PR proposes to add more flexible `getAllWithPrefix` for `SparkConf`.
We need to set some config related to S3 for our inner Spark.
The requirements are replacing the prefix `spark.fs.s3a` with new prefix
`spark.hadoop.fs.s3a` The implementation of the function show below.
```
private def setS3Configs(conf: SparkConf): Unit = {
val S3A_PREFIX = "spark.fs.s3a"
val SPARK_HADOOP_S3A_PREFIX = "spark.hadoop.fs.s3a"
val s3aConf = conf.getAllWithPrefix(S3A_PREFIX)
s3aConf
.foreach(
confPair => {
val keyWithoutPrefix = confPair._1
val oldKey = S3A_PREFIX + keyWithoutPrefix
val newKey = SPARK_HADOOP_S3A_PREFIX + keyWithoutPrefix
val value = confPair._2
(newKey, value)
})
}
```
Because `getAllWithPrefix` truncated the suffix, developers must concat the
suffix and prefix to restore the original key.
The new `getAllWithPrefix` increases the flexibility developers could
customize the function as they want.
After this change, the code show above could be improved as follows.
```
private def setS3Configs(conf: SparkConf): Unit = {
val S3A_PREFIX = "spark.fs.s3a"
val SPARK_HADOOP_S3A_PREFIX = "spark.hadoop.fs.s3a"
val f = (k: String) => {
val keyWithoutPrefix = k.substring(S3A_PREFIX.length)
"spark.hadoop.fs.s3a" + keyWithoutPrefix
}
conf.getAllWithPrefix("spark.fs.s3a", f)
}
```
### Why are the changes needed?
The new API `getAllWithPrefix` could improve the flexibility for `SparkConf`.
### Does this PR introduce _any_ user-facing change?
'No'.
New API.
### How was this patch tested?
GA tests.
### Was this patch authored or co-authored using generative AI tooling?
'No'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
