[ 
https://issues.apache.org/jira/browse/SPARK-60653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-60653:
-----------------------------------
    Labels: pull-request-available  (was: )

> pandas-on-Spark np.lcm declares a double result and fails for an int64 value 
> above 2**53
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-60653
>                 URL: https://issues.apache.org/jira/browse/SPARK-60653
>             Project: Spark
>          Issue Type: Bug
>          Components: Pandas API on Spark, PySpark
>    Affects Versions: 4.2.0
>         Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
> Python: 3.11.17
> pyspark 4.2.0
> findspark 2.0.1
> pandas 3.0.6
> numpy 2.4.6
>            Reporter: asddfl
>            Priority: Critical
>              Labels: pull-request-available
>
> When using the pandas API on Spark (pyspark.pandas), np.lcm fails for the 
> valid int64 value 2**53 + 1. The same operation on a Pandas Series returns 
> the integer unchanged.
> The failure occurs while converting the integer result to Arrow double in the 
> Python worker. This input and its least common multiple are both within the 
> int64 range, but 2**53 + 1 cannot be represented exactly as float64.
> {code:python}
> import pandas as pd
> import numpy as np
> from pyspark.sql import SparkSession
> import pyspark.pandas as ps
> t0 = pd.DataFrame({'c0': [2**53 + 1]})
> print("Pandas:", flush=True)
> result = np.lcm(t0['c0'], t0['c0'])
> print(result, flush=True)
> spark = (
>     SparkSession.builder
>     .master("local[2]")
>     .config("spark.sql.ansi.enabled", "false")
>     .config("spark.sql.shuffle.partitions", "2")
>     .getOrCreate()
> )
> t0 = ps.DataFrame({'c0': [2**53 + 1]})
> print("PySpark Pandas:", flush=True)
> result = np.lcm(t0['c0'], t0['c0'])
> print(result, flush=True)
> spark.stop()
> {code}
> {code:bash}
> Pandas:
> 0    9007199254740993
> Name: c0, dtype: int64
> PySpark Pandas:
> Traceback (most recent call last):
>   File "/test.py", line 26, in <module>
>     print(result, flush=True)
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/series.py", line 
> 7481, in __repr__
>     pser = 
> self._psdf._get_or_create_repr_pandas_cache(max_display_count)[self.name]
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py", line 
> 13988, in _get_or_create_repr_pandas_cache
>     self, "_repr_pandas_cache", {n: self.head(n + 1)._to_internal_pandas()}
>                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py", line 
> 13983, in _to_internal_pandas
>     return self._internal.to_pandas_frame
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/utils.py", line 
> 608, in wrapped_lazy_property
>     setattr(self, attr_name, fn(self))
>                              ^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/internal.py", 
> line 1080, in to_pandas_frame
>     pdf = sdf._to_pandas(pandasStructHandlingMode="row")
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
>  line 1974, in _to_pandas
>     return PandasConversionMixin._to_pandas(self, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/pandas/conversion.py",
>  line 367, in _to_pandas
>     batches = self._collect_as_arrow(
>               ^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/pandas/conversion.py",
>  line 537, in _collect_as_arrow
>     with unwrap_spark_exception():
>   File "/opt/conda/envs/lib/python3.11/contextlib.py", line 158, in __exit__
>     self.gen.throw(typ, value, traceback)
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
>  line 280, in unwrap_spark_exception
>     raise converted from None
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
>  line 410, in convert_column
>   File "pyarrow/array.pxi", line 1341, in pyarrow.lib.Array.from_pandas
>   File "pyarrow/array.pxi", line 414, in pyarrow.lib.array
>   File "pyarrow/array.pxi", line 97, in pyarrow.lib._ndarray_to_array
>   File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
> pyspark.errors.exceptions.captured.PythonException: An exception was thrown 
> from the Python worker:
> Traceback (most recent call last):
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
>  line 410, in convert_column
>     return pa.Array.from_pandas(
>            ^^^^^^^^^^^^^^^^^^^^^
>   File "pyarrow/array.pxi", line 1341, in pyarrow.lib.Array.from_pandas
>   File "pyarrow/array.pxi", line 414, in pyarrow.lib.array
>   File "pyarrow/array.pxi", line 97, in pyarrow.lib._ndarray_to_array
>   File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Integer value 9007199254740993 not in range: 
> -9007199254740992 to 9007199254740992
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
>  line 415, in convert_column
>     return pa.Array.from_pandas(series, mask=mask).cast(
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "pyarrow/array.pxi", line 1217, in pyarrow.lib.Array.cast
>   File "/opt/conda/envs/lib/python3.11/site-packages/pyarrow/compute.py", 
> line 415, in cast
>     return call_function("cast", [arr], options, memory_pool)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "pyarrow/_compute.pyx", line 610, in pyarrow._compute.call_function
>   File "pyarrow/_compute.pyx", line 405, in pyarrow._compute.Function.call
>   File "pyarrow/error.pxi", line 155, in 
> pyarrow.lib.pyarrow_internal_check_status
>   File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Integer value 9007199254740993 not in range: 
> -9007199254740992 to 9007199254740992
> The above exception was the direct cause of the following exception:
> Traceback (most recent call last):
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py",
>  line 3622, in main
>     process()
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py",
>  line 3612, in process
>     serializer.dump_stream(out_iter, outfile)
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
>  line 546, in dump_stream
>     return ArrowStreamSerializer.dump_stream(self, batches, stream)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
>  line 146, in dump_stream
>     for batch in iterator:
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
>  line 168, in _write_stream_start
>     first = next(batch_iterator, None)
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
>  line 543, in <genexpr>
>     (create_batch(_normalize_packed(packed)) for packed in iterator),
>      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
>  line 529, in create_batch
>     return PandasToArrowConversion.convert(
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
>  line 442, in convert
>     arrays = [convert_column(col, field) for col, field in zip(columns, 
> schema.fields)]
>              
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
>  line 442, in <listcomp>
>     arrays = [convert_column(col, field) for col, field in zip(columns, 
> schema.fields)]
>               ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
>  line 440, in convert_column
>     raise PySparkValueError(error_msg) from e
> pyspark.errors.exceptions.base.PySparkValueError: Failed to convert the value 
> of the column '_0' with type 'int64' to Arrow type 'double'. It can be caused 
> by overflows or other unsafe conversions warned by Arrow. Arrow safe type 
> check can be disabled by using SQL config 
> `spark.sql.execution.pandas.convertToArrowArraySafely`.
> {code}
> *Expected behavior*
> np.lcm applied to two Series containing 9007199254740993 should return an 
> integer Series containing 9007199254740993, matching Pandas without an unsafe 
> conversion to double.
> *Actual behavior*
> The computation raises pyspark.errors.exceptions.captured.PythonException. 
> Its worker traceback contains pyarrow.lib.ArrowInvalid: Integer value 
> 9007199254740993 not in range: -9007199254740992 to 9007199254740992, 
> followed by a failure to convert an int64 column to Arrow double.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to