[ https://issues.apache.org/jira/browse/SPARK-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108762#comment-15108762 ]
Apache Spark commented on SPARK-12926: -------------------------------------- User 'tejasapatil' has created a pull request for this issue: https://github.com/apache/spark/pull/10849 > SQLContext to disallow users passing non-sql configs > ---------------------------------------------------- > > Key: SPARK-12926 > URL: https://issues.apache.org/jira/browse/SPARK-12926 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 1.6.0 > Reporter: Tejas Patil > Priority: Trivial > > Users unknowingly try to set core Spark configs in sqlContext but later > realise that it didn't work. > {color:red} > scala> sqlContext.sql("SET spark.shuffle.memoryFraction=0.4") > res3: org.apache.spark.sql.DataFrame = [key: string, value: string] > scala> sqlContext.getConf.get("spark.shuffle.memoryFraction") > java.util.NoSuchElementException: spark.shuffle.memoryFraction > at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193) > at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.SparkConf.get(SparkConf.scala:193) > {color} > We could do this: > - for configs starting with "spark.", allow only if it starts with > "spark.sql.". Otherwise disallow. > - allow anything else. > This will be a simple change in SqlConf : > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala#L621 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org