Yicong-Huang commented on code in PR #56402:
URL: https://github.com/apache/spark/pull/56402#discussion_r3399626490
##########
python/pyspark/sql/tests/coercion/test_pandas_udf_input_type.py:
##########
@@ -251,14 +263,6 @@ def _compare_or_generate_golden(self, golden_file,
test_name):
golden = None
if not generating:
golden = self.load_golden_csv(golden_csv)
- # Pandas >= 3.0 reports the dedicated 'str' dtype for string
columns,
- # whereas earlier versions report 'object'. Patch the in-memory
golden
- # so the same file works under both versions.
- if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
- str_rows = golden["Spark Type"] == "string"
- golden.loc[str_rows, "Python Type"] = golden.loc[
- str_rows, "Python Type"
- ].str.replace("'object'", "'str'")
Review Comment:
The good point of those if else is easier for use to understand the
difference between pandas 2 and pandas 3. I see the benefit of using major
version specific golden file, but no we lose this "diff" view in source code.
Maybe we should find another way to track the known pandas 2 and pandas 3
diffs, some options I can think of:
- keep a md file to log for diff;
- recommend to use file diff on pandas2_golden_file and
pandas3_golden_file;
##########
python/pyspark/sql/tests/coercion/test_pandas_udf_return_type.py:
##########
@@ -144,7 +157,7 @@ def test_str_repr(self):
def test_pandas_return_type_coercion_vanilla(self):
self._run_pandas_udf_return_type_coercion(
- golden_file=f"{self.prefix}_base",
+ golden_file=f"{self.pandas_dir}/{self.prefix}_base",
Review Comment:
nit: `/` only works for Unix right? shall we use Path lib to construct the
file name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
