zhengruifeng commented on code in PR #56402:
URL: https://github.com/apache/spark/pull/56402#discussion_r3400068486
##########
python/pyspark/sql/tests/coercion/test_pandas_udf_input_type.py:
##########
@@ -251,14 +263,6 @@ def _compare_or_generate_golden(self, golden_file,
test_name):
golden = None
if not generating:
golden = self.load_golden_csv(golden_csv)
- # Pandas >= 3.0 reports the dedicated 'str' dtype for string
columns,
- # whereas earlier versions report 'object'. Patch the in-memory
golden
- # so the same file works under both versions.
- if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
- str_rows = golden["Spark Type"] == "string"
- golden.loc[str_rows, "Python Type"] = golden.loc[
- str_rows, "Python Type"
- ].str.replace("'object'", "'str'")
Review Comment:
Good point. With the goldens split per version, the "diff view" moves from
source code into the files themselves: the file names are identical across
`pandas_2/` and `pandas_3/`, so `diff pandas_2/golden_X.csv
pandas_3/golden_X.csv` shows it directly — and it is also easy to have an AI
assistant compare the two directories and summarize the behavior differences
(the summary below was produced that way). I would lean on that rather than a
hand-maintained md log, which can drift from the actual files.
For the record, the complete pandas 2 -> 3 delta across the four golden
pairs:
**`golden_pandas_udf_return_type_coercion_base`**
- column headers (test value reprs): naive and tz-aware datetime ndarrays
`datetime64[ns]` -> `datetime64[us]`; `Categorical` categories dtype `object`
-> `str`
- cells: str list -> `decimal(10,0)` flips from error (`X`) to
`[Decimal('12'), Decimal('34')]`; `Timedelta` list -> `bigint` values scale ns
-> us (`86400000000000` -> `86400000000`)
**`golden_pandas_udf_input_type_coercion_base`**
- `string_values` / `string_null` rows: the Series dtype seen by the UDF for
string columns is `'object'` -> `'str'`
**`golden_python_udf_input_type_coercion_with_arrow_and_pandas`**
- `string_null` row: pandas 2 round-trips `[None, 'test']` intact; pandas 3
turns `None` into `nan`, recorded as `✗ Output ['nan', 'test'] != Input [None,
'test']`
**`golden_python_udf_return_type_coercion_with_arrow_and_pandas`**
- byte-identical (forked only because the legacy conversion path routes
through pandas)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
