[ https://issues.apache.org/jira/browse/SPARK-16272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354211#comment-15354211 ]
Marcelo Vanzin commented on SPARK-16272: ---------------------------------------- BTW, I have the code mostly ready, just needs more testing and some cleanup. I'll hook up the config variable I mentioned above as an example, since that's the one that I ran into, and afterwards we can modify other configs to also use the functionality. > Allow configs to reference other configs, env and system properties > ------------------------------------------------------------------- > > Key: SPARK-16272 > URL: https://issues.apache.org/jira/browse/SPARK-16272 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Reporter: Marcelo Vanzin > Priority: Minor > > Currently, Spark's configuration is static; it is whatever is written to the > config file, with some rare exceptions (such as some YARN code that does > expansion of Hadoop configuration). > But there are a few use cases that don't work well in that situation. For > example, consider {{spark.sql.hive.metastore.jars}}. It references a list of > paths containing the classpath for accessing Hive's metastore. If you're > launching an application in cluster mode, it means that whatever is in the > configuration of the edge node needs to match the configuration of the random > node in the cluster where the driver will actually run. > This would be easily solved if there was a way to reference system properties > or env variables; for example, when YARN launches a container, a bunch of env > variables are set, which could be used to modify that path to match the > correct location on the node. > So I'm proposing a change where config properties can opt-in to use this > variable expansion feature; it's opt-in to avoid breaking existing code (who > knows) and to avoid the extra cost of doing the variable expansion of every > config read. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org