allisonwang-db commented on code in PR #41606:
URL: https://github.com/apache/spark/pull/41606#discussion_r1254998841
##########
python/pyspark/testing/utils.py:
##########
@@ -209,3 +219,131 @@ def check_error(
self.assertEqual(
expected, actual, f"Expected message parameters was '{expected}',
got '{actual}'"
)
+
+
+def assertDataFrameEqual(
+ df: DataFrame, expected: Union[DataFrame, List[Row]], ignore_row_order:
bool = True
+):
+ """
+ A util function to assert equality between DataFrames `df` and `expected`,
with
+ optional parameter `ignore_row_order`.
Review Comment:
Nit: maybe we should name this `check_row_order`, since the function by
default does not check the row order. If users want to enforce this, they can
specify this `check_row_order=True`. WDYT?
##########
python/pyspark/testing/utils.py:
##########
@@ -209,3 +219,131 @@ def check_error(
self.assertEqual(
expected, actual, f"Expected message parameters was '{expected}',
got '{actual}'"
)
+
+
+def assertDataFrameEqual(
+ df: DataFrame, expected: Union[DataFrame, List[Row]], ignore_row_order:
bool = True
+):
+ """
+ A util function to assert equality between DataFrames `df` and `expected`,
with
+ optional parameter `ignore_row_order`.
+
Review Comment:
.. versionadded:: 3.5.0
##########
python/pyspark/testing/utils.py:
##########
@@ -209,3 +219,131 @@ def check_error(
self.assertEqual(
expected, actual, f"Expected message parameters was '{expected}',
got '{actual}'"
)
+
+
+def assertDataFrameEqual(
+ df: DataFrame, expected: Union[DataFrame, List[Row]], ignore_row_order:
bool = True
+):
+ """
+ A util function to assert equality between DataFrames `df` and `expected`,
with
+ optional parameter `ignore_row_order`.
+
+ For float values, assert approximate equality (1e-5 by default).
+
+ Parameters
+ ----------
+ df : DataFrame
+ expected : DataFrame or List of Row
+ ignore_row_order: bool, default True
Review Comment:
Let's add more details in the docstring.
```
Parameters
----------
df : DataFrame
The DataFrame that is being compared or tested.
expected : DataFrame or list of Row
The expected result of the operation, for comparison with the actual
result.
check_row_order : bool, optional
A flag indicates whether the order of rows should be considered in the
comparison. If set to `False` (default), the row order is not taken into
account. If set to `True`, the order of rows is important and will be checked
during comparison.
Examples
--------
<add some examples here>
assertDataFrameEqual(df1, df2) # good
assertDataFrameEqual(df1, df2) # bad
***** error message here
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
