HyukjinKwon commented on code in PR #41947:
URL: https://github.com/apache/spark/pull/41947#discussion_r1262425446
##########
python/pyspark/testing/utils.py:
##########
@@ -250,32 +268,29 @@ def assertDataFrameEqual(df: DataFrame, expected:
DataFrame, check_row_order: bo
.config("spark.some.config.option", "some-value").getOrCreate()
>>> df1 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)],
schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)],
schema=["id", "amount"])
- >>> assertDataFrameEqual(df1, df2) # pass
- >>> df1 = spark.createDataFrame(data=[("1", 1000.00), ("2", 3000.00),
("3", 2000.00)], \
- schema=["id", "amount"])
- >>> df2 = spark.createDataFrame(data=[("1", 1001.00), ("2", 3000.00),
("3", 2003.00)], \
- schema=["id", "amount"])
- >>> assertDataFrameEqual(df1, df2) # fail # doctest:
+IGNORE_EXCEPTION_DETAIL
+ >>> assertDataFrameEqual(df1, df2) # DataFrames are equal
+ >>> df1 = spark.createDataFrame(data=[("1", 0.1), ("2", 3.23)],
schema=["id", "amount"])
+ >>> df2 = spark.createDataFrame(data=[("1", 0.109), ("2", 3.23)],
schema=["id", "amount"])
+ >>> assertDataFrameEqual(df1, df2, rtol=1e-1) # DataFrames are approx
equal by rtol
+ >>> df1 = spark.createDataFrame(data=[("1", 1000.00), ("2", 3000.00),
("3", 2000.00)],
+ ... schema=["id", "amount"])
+ >>> df2 = spark.createDataFrame(data=[("1", 1001.00), ("2", 3000.00),
("3", 2003.00)],
+ ... schema=["id", "amount"])
+ >>> assertDataFrameEqual(df1, df2) # doctest: +IGNORE_EXCEPTION_DETAIL
Review Comment:
Can we fix as follows?
```suggestion
>>> assertDataFrameEqual(df1, df2) # DataFrames are equal
explainataion .. blah blah..
>>> df1 = spark.createDataFrame(data=[("1", 0.1), ("2", 3.23)],
schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 0.109), ("2", 3.23)],
schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2, rtol=1e-1) #
Check equality of float .. blah blah..
>>> df1 = spark.createDataFrame(
... data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)],
schema=["id", "amount"])
>>> df2 = spark.createDataFrame(
... data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)],
schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2) # doctest: +IGNORE_EXCEPTION_DETAIL
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
