[
https://issues.apache.org/jira/browse/SPARK-7961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-7961:
-------------------------------
Description:
Right now, we don't validate config values and as a result will throw
exceptions when queries or DataFrame operations are run.
Imagine if one user sets config variable "spark.sql.retainGroupColumns"
(requires "true", "false") to "hello". The set action itself will complete
fine. When another user runs a query, it will throw the following exception:
{code}
java.lang.IllegalArgumentException: For input string: "hello"
at
scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:238)
at
scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:226)
at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:31)
at
org.apache.spark.sql.SQLConf.dataFrameRetainGroupColumns(SQLConf.scala:265)
at org.apache.spark.sql.GroupedData.toDF(GroupedData.scala:74)
at org.apache.spark.sql.GroupedData.agg(GroupedData.scala:227)
{code}
This is highly confusing. We should redesign SQLConf to validate data input at
set time (during setConf call).
was:
Right now, we don't validate config values and as a result will throw
exceptions when queries or DataFrame operations are run.
Imagine if one user sets config variable "spark.sql.retainGroupColumns"
(requires "true", "false") to "hello". The set action itself will complete
fine. When another user runs a query, it will throw the following exception:
{code}
java.lang.IllegalArgumentException: For input string: "hello"
at
scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:238)
at
scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:226)
at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:31)
at
org.apache.spark.sql.SQLConf.dataFrameRetainGroupColumns(SQLConf.scala:265)
at org.apache.spark.sql.GroupedData.toDF(GroupedData.scala:74)
at org.apache.spark.sql.GroupedData.agg(GroupedData.scala:227)
{code}
> Redesign SQLConf for better error message reporting
> ---------------------------------------------------
>
> Key: SPARK-7961
> URL: https://issues.apache.org/jira/browse/SPARK-7961
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Reynold Xin
> Priority: Blocker
>
> Right now, we don't validate config values and as a result will throw
> exceptions when queries or DataFrame operations are run.
> Imagine if one user sets config variable "spark.sql.retainGroupColumns"
> (requires "true", "false") to "hello". The set action itself will complete
> fine. When another user runs a query, it will throw the following exception:
> {code}
> java.lang.IllegalArgumentException: For input string: "hello"
> at
> scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:238)
> at
> scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:226)
> at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:31)
> at
> org.apache.spark.sql.SQLConf.dataFrameRetainGroupColumns(SQLConf.scala:265)
> at org.apache.spark.sql.GroupedData.toDF(GroupedData.scala:74)
> at org.apache.spark.sql.GroupedData.agg(GroupedData.scala:227)
> {code}
> This is highly confusing. We should redesign SQLConf to validate data input
> at set time (during setConf call).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
