HyukjinKwon commented on a change in pull request #27565:
[SPARK-30791][SQL][PYTHON] Add 'sameSemantics' and 'sementicHash' methods in
Dataset
URL: https://github.com/apache/spark/pull/27565#discussion_r379991667
##########
File path: python/pyspark/sql/dataframe.py
##########
@@ -2153,6 +2153,58 @@ def transform(self, func):
"should have been DataFrame." %
type(result)
return result
+ @since(3.1)
+ def sameSemantics(self, other):
+ """
+ Returns `True` when the logical query plans inside both
:class:`DataFrame`\\s are equal and
+ therefore return same results.
+
+ .. note:: The equality comparison here is simplified by tolerating the
cosmetic differences
+ such as attribute names.
+
+ .. note:: This API can compare both :class:`DataFrame`\\s very fast
but can still return
+ `False` on the :class:`DataFrame` that return the same results,
for instance, from
+ different plans. Such false negative semantic can be useful when
caching as an example.
+
+ >>> df1 = spark.range(100)
+ >>> df2 = spark.range(100)
+ >>> df3 = spark.range(100)
+ >>> df4 = spark.range(100)
+ >>> df1.withColumn("col1", df1.id *
2).sameSemantics(df2.withColumn("col1", df2.id * 2))
+ True
+ >>> df1.withColumn("col1", df1.id *
2).sameSemantics(df3.withColumn("col1", df3.id + 2))
+ False
+ >>> df1.withColumn("col1", df1.id *
2).sameSemantics(df4.withColumn("col0", df4.id * 2))
+ True
+ """
Review comment:
nit:
```python
>>> df1 = spark.range(10)
>>> df2 = spark.range(10)
>>> df1.withColumn("col1", df1.id *
2).sameSemantics(df2.withColumn("col1", df2.id * 2))
True
>>> df1.withColumn("col1", df1.id *
2).sameSemantics(df2.withColumn("col1", df2.id + 2))
False
>>> df1.withColumn("col1", df1.id *
2).sameSemantics(df2.withColumn("col0", df2.id * 2))
True
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
