Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/10435#discussion_r48342878
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala
---
@@ -114,6 +114,7 @@ private[sql] object StatFunctions extends Logging {
if (element == null) "null" else element.toString
}
// get the distinct values of column 2, so that we can make them the
column names
+ // ???
val distinctCol2: Map[Any, Int] =
counts.map(e => cleanElement(e.get(1))).distinct.zipWithIndex.toMap
--- End diff --
The `counts` is the result of aggregation without sort, so the order of
distinct values of column 2 is non-deterministic. This means, when we use
`distinctCol2` to generate the field names of output schema, the order of field
names is also non-deterministic.
When I changed the hash function in `Exchange`, one crosstable test failed
because the field names don't match. I simply updated that test, need someone
to confirm this is the right thing to do.
cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
