Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/15959#discussion_r89053783
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala
---
@@ -100,99 +100,30 @@ object AnalyzeColumnCommand extends Logging {
exprOption.getOrElse(throw new AnalysisException(s"Invalid column
name: $col."))
}).toSeq
+ // Make sure the column types are supported for stats gathering.
+ attributesToAnalyze.foreach { attr =>
+ if (!ColumnStat.supportsType(attr.dataType)) {
+ throw new AnalysisException(
+ s"Data type ${attr.dataType.simpleString} for column
${attr.name} is not supported " +
--- End diff --
Yes. The command can be ran in a context of a pipeline with other commands.
Say a job is writing to `n` tables and then populating stats for each of them.
In case one of the queries to populate the stats fails, user would have to dig
through the logs to find out which table caused the problem. If the exception
message has ample information, one can simply look at the failure message of
the container where the driver ran without needing to scroll through logs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
