HeartSaVioR commented on a change in pull request #28830:
URL: https://github.com/apache/spark/pull/28830#discussion_r439904675
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2541,7 +2542,20 @@ class Dataset[T] private[sql](
def dropDuplicates(colNames: Seq[String]): Dataset[T] = withTypedPlan {
val resolver = sparkSession.sessionState.analyzer.resolver
val allColumns = queryExecution.analyzed.output
- val groupCols = colNames.distinct.flatMap { (colName: String) =>
+ // SPARK-31990: We must preserve the input order of `colNames` because of
the compatibility
+ // issue (the Streaming's state store depends on the `groupCols` order).
+ val orderPreservingDistinctColNames = {
+ val nameSeen = mutable.Set[String]()
Review comment:
I think Scala wouldn't change the implementation unless there's strong
reason to do so, but in Scala 2.13 they removed the orderness guarantee on the
method doc in `distinct`. Till Scala 2.12 it clearly explains the return value
as preserving the "first occurrence", and the explanation no longer exists in
Scala 2.13.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
