Reynold Xin created SPARK-7961:
----------------------------------
Summary: Redesign SQLConf to report better error messages
Key: SPARK-7961
URL: https://issues.apache.org/jira/browse/SPARK-7961
Project: Spark
Issue Type: Improvement
Components: SQL
Reporter: Reynold Xin
Priority: Blocker
Right now, we don't validate config values and as a result will throw
exceptions when queries or DataFrame operations are run.
Imagine if one user sets config variable "spark.sql.retainGroupColumns"
(requires "true", "false") to "hello". The set action itself will complete
fine. When another user runs a query, it will throw the following exception:
{code}
java.lang.IllegalArgumentException: For input string: "hello"
at
scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:238)
at
scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:226)
at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:31)
at
org.apache.spark.sql.SQLConf.dataFrameRetainGroupColumns(SQLConf.scala:265)
at org.apache.spark.sql.GroupedData.toDF(GroupedData.scala:74)
at org.apache.spark.sql.GroupedData.agg(GroupedData.scala:227)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
