Spenserrrr commented on code in PR #57939: URL: https://github.com/apache/spark/pull/57939#discussion_r3765122665
########## python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py: ########## @@ -0,0 +1,340 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +""" +Tests for PyArrow Table.to_pandas() using golden file comparison. + +Unlike Array/ChunkedArray.to_pandas() (which returns a Series and is covered by +test_pyarrow_arrow_to_pandas_{default,non_default}.py), Table.to_pandas() returns a +DataFrame. Its per-column conversion matches the Array tests, so this file pins the +genuinely Table-specific behavior instead: multi-column DataFrame assembly and the +empty-table edges (0 columns / 0 rows). Spark calls Table.to_pandas() at +python/pyspark/sql/pandas/conversion.py:255 (the 0-column path) and, in Spark Connect, +at python/pyspark/sql/connect/client/core.py:1423 (a bare whole-Table conversion). + +## Golden File Cell Format + +Each cell uses the value@type format: +- pyarrow Table: "{col: [val1, val2, None], ...}@Table[name: type, ...]" +- pandas DataFrame: "{col: [values], ...}@Dataframe[name dtype, ...]" +- Error: "ERR@ExceptionClassName" + +Values are formatted via tolist() for stable, Python-native representation. + +## Regenerating Golden Files + +Set SPARK_GENERATE_GOLDEN_FILES=1 before running: + + SPARK_GENERATE_GOLDEN_FILES=1 python -m pytest \\ + python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py +""" + +import datetime +import unittest + +from pyspark.loose_version import LooseVersion +from pyspark.testing.utils import ( + have_pyarrow, + have_pandas, + pyarrow_requirement_message, + pandas_requirement_message, +) +from pyspark.testing.goldenutils import GoldenFileTestMixin + +if have_pandas: + import pandas as pd +if have_pyarrow: + import pyarrow as pa + + +class _PyArrowTableToPandasTestBase(GoldenFileTestMixin, unittest.TestCase): + """ + Shared machinery for pa.Table.to_pandas() golden file tests. + + Holds the conversion helper and the source-table inventory, split into group + methods that these and the (temporal-flag) tests reuse. Defines no ``test_*`` of + its own. + """ + + @staticmethod + def _repr_dataframe(df) -> str: + """ + Format a pandas DataFrame result as a golden-file cell: per-column tolist() + for a stable, Python-native representation, mirroring goldenutils' + ``repr_pandas_series_value`` for the Series case. + + This deliberately does NOT go through ``repr_value``/``repr_pandas_value`` + (which use ``to_json``). With the default date_as_object=True a far-future + date (year 9999) comes back as an object column of Python ``datetime.date`` + objects, and ``DataFrame.to_json`` overflows converting each to an epoch + nanosecond int (OverflowError); tolist() returns the objects as-is, so + nothing overflows. + """ + body = str({name: col.tolist() for name, col in df.items()}).replace("\n", " ") + schema = ", ".join(f"{t} {d.name}" for t, d in df.dtypes.items()) + return f"{body}@Dataframe[{schema}]" + + def _to_pandas_cell(self, table, **to_pandas_kwargs) -> str: + """ + Convert ``table`` via ``to_pandas(**to_pandas_kwargs)`` and format the result + as a golden-file cell, returning ``ERR@<ExceptionClass>`` if it raises. + """ + try: + return self._repr_dataframe(table.to_pandas(**to_pandas_kwargs)) Review Comment: Thanks for catching this and I just fixed it. The same both-wrapped pattern exists in the already-merged Array to_pandas tests as well and I'll unify this fix across them in a follow-up PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
