[
https://issues.apache.org/jira/browse/SPARK-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191461#comment-16191461
]
Sean Owen commented on SPARK-22201:
-----------------------------------
This is just a duplicate of SPARK-16468
What API are you referring to? the scaladoc says it computes stats for numeric
and string types.
It's supposed to return stats on strings. They have an ordering, so max/min is
coherent.
> Dataframe describe includes string columns
> ------------------------------------------
>
> Key: SPARK-22201
> URL: https://issues.apache.org/jira/browse/SPARK-22201
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: cold gin
>
> As per the api documentation, the default no-arg Dataframe describe()
> function should only include numerical column types, but it is including
> String types as well. This creates unusable statistical results (for example,
> max returns "V8903" for one of the string columns in my dataset).
> There also appears to be several related issues to this:
> https://issues.apache.org/jira/browse/SPARK-16468
> https://issues.apache.org/jira/browse/SPARK-16429
> But SPARK-16429 does not make sense with what the default api says, and only
> Int, Double, etc (numeric) columns should be included when generating the
> statistics.
> Perhaps this reveals the need for a new function to produce stats that make
> sense only for string columns, or else an additional parameter to describe()
> to filter in/out certain column types?
> In summary, the *default* describe api behavior (no arg behavior) should not
> include string columns.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
