viirya commented on a change in pull request #28996:
URL: https://github.com/apache/spark/pull/28996#discussion_r453054212
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2030,7 +2030,25 @@ class Dataset[T] private[sql](
* @group typedrel
* @since 2.3.0
*/
- def unionByName(other: Dataset[T]): Dataset[T] = withSetOperator {
+ def unionByName(other: Dataset[T]): Dataset[T] = unionByName(other, false)
+
+ /**
+ * Returns a new Dataset containing union of rows in this Dataset and
another Dataset.
+ *
+ * This is different from both `UNION ALL` and `UNION DISTINCT` in SQL. To
do a SQL-style set
+ * union (that does deduplication of elements), use this function followed
by a [[distinct]].
Review comment:
I read `To do a SQL-style set union`, it sounds like if you add
`distinct`, you will get a SQL-style union. But it behaves different to SQL
union at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
