[
https://issues.apache.org/jira/browse/SPARK-7961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565803#comment-14565803
]
Reynold Xin commented on SPARK-7961:
------------------------------------
One other thing that'd be nice as part of this redesign is to be able to
support some sort of list all config options, their values, and the
documentation for them in a sql command. We should mark certain config options
as internal so they don't get displayed.
> Redesign SQLConf for better error message reporting
> ---------------------------------------------------
>
> Key: SPARK-7961
> URL: https://issues.apache.org/jira/browse/SPARK-7961
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Reynold Xin
> Priority: Blocker
>
> Right now, we don't validate config values and as a result will throw
> exceptions when queries or DataFrame operations are run.
> Imagine if one user sets config variable "spark.sql.retainGroupColumns"
> (requires "true", "false") to "hello". The set action itself will complete
> fine. When another user runs a query, it will throw the following exception:
> {code}
> java.lang.IllegalArgumentException: For input string: "hello"
> at
> scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:238)
> at
> scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:226)
> at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:31)
> at
> org.apache.spark.sql.SQLConf.dataFrameRetainGroupColumns(SQLConf.scala:265)
> at org.apache.spark.sql.GroupedData.toDF(GroupedData.scala:74)
> at org.apache.spark.sql.GroupedData.agg(GroupedData.scala:227)
> {code}
> This is highly confusing. We should redesign SQLConf to validate data input
> at set time (during setConf call).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
