nsuke commented on code in PR #41168:
URL: https://github.com/apache/spark/pull/41168#discussion_r1194385787
##########
core/src/main/scala/org/apache/spark/SparkConf.scala:
##########
@@ -403,7 +403,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable
with Logging with Seria
*/
def getAllWithPrefix(prefix: String): Array[(String, String)] = {
Review Comment:
Wouldn't that mean the current inconsistent behavior would only be partially
resolved by this PR? We may want to be more explicit about when to substitute
and when not to.
A few interesting cases are `spark.hadoop.*` and `spark.hive.*`:
https://github.com/apache/spark/blob/c8e85eab3fca0e4e5f4bdf9d1d6d1702ecf3fd07/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L506
https://github.com/apache/spark/blob/c8e85eab3fca0e4e5f4bdf9d1d6d1702ecf3fd07/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala#L534
these particular cases may be easily covered by switching to
`getAllWithPrefix` though. (assuming that the expected behavior is to do
substitution here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
