Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9300#discussion_r43170048
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -360,6 +357,48 @@ class Dataset[T] private[sql](
*/
def subtract(other: Dataset[T]): Dataset[T] = withPlan[T](other)(Except)
+ /* ****** *
+ * Joins *
+ * ****** */
+
+ /**
+ * Joins this [[Dataset]] returning a [[Tuple2]] for each pair where
`condition` evaluates to
+ * true.
+ *
+ * This is similar to the relation `join` function with one important
difference in the
+ * result schema. Since `joinWith` preserves objects present on either
side of the join, the
+ * result schema is similarly nested into a tuple under the column names
`_1` and `_2`.
+ *
+ * This type of join can be useful both for preserving type-safety with
the original object
+ * types as well as working with relational data where either side of
the join has column
+ * names in common.
+ */
+ def joinWith[U](other: Dataset[U], condition: Column): Dataset[(T, U)] =
{
+ val left = this.logicalPlan
+ val right = other.logicalPlan
+
+ val leftData = this.encoder match {
+ case e if e.flat => Alias(left.output.head, "_1")()
--- End diff --
Do we need to do anything special to handle cases that these tables have
common name columns?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
