[
https://issues.apache.org/jira/browse/SPARK-60653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60653:
-----------------------------------
Labels: pull-request-available (was: )
> pandas-on-Spark np.lcm declares a double result and fails for an int64 value
> above 2**53
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-60653
> URL: https://issues.apache.org/jira/browse/SPARK-60653
> Project: Spark
> Issue Type: Bug
> Components: Pandas API on Spark, PySpark
> Affects Versions: 4.2.0
> Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
> Python: 3.11.17
> pyspark 4.2.0
> findspark 2.0.1
> pandas 3.0.6
> numpy 2.4.6
> Reporter: asddfl
> Priority: Critical
> Labels: pull-request-available
>
> When using the pandas API on Spark (pyspark.pandas), np.lcm fails for the
> valid int64 value 2**53 + 1. The same operation on a Pandas Series returns
> the integer unchanged.
> The failure occurs while converting the integer result to Arrow double in the
> Python worker. This input and its least common multiple are both within the
> int64 range, but 2**53 + 1 cannot be represented exactly as float64.
> {code:python}
> import pandas as pd
> import numpy as np
> from pyspark.sql import SparkSession
> import pyspark.pandas as ps
> t0 = pd.DataFrame({'c0': [2**53 + 1]})
> print("Pandas:", flush=True)
> result = np.lcm(t0['c0'], t0['c0'])
> print(result, flush=True)
> spark = (
> SparkSession.builder
> .master("local[2]")
> .config("spark.sql.ansi.enabled", "false")
> .config("spark.sql.shuffle.partitions", "2")
> .getOrCreate()
> )
> t0 = ps.DataFrame({'c0': [2**53 + 1]})
> print("PySpark Pandas:", flush=True)
> result = np.lcm(t0['c0'], t0['c0'])
> print(result, flush=True)
> spark.stop()
> {code}
> {code:bash}
> Pandas:
> 0 9007199254740993
> Name: c0, dtype: int64
> PySpark Pandas:
> Traceback (most recent call last):
> File "/test.py", line 26, in <module>
> print(result, flush=True)
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/series.py", line
> 7481, in __repr__
> pser =
> self._psdf._get_or_create_repr_pandas_cache(max_display_count)[self.name]
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py", line
> 13988, in _get_or_create_repr_pandas_cache
> self, "_repr_pandas_cache", {n: self.head(n + 1)._to_internal_pandas()}
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py", line
> 13983, in _to_internal_pandas
> return self._internal.to_pandas_frame
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/utils.py", line
> 608, in wrapped_lazy_property
> setattr(self, attr_name, fn(self))
> ^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/internal.py",
> line 1080, in to_pandas_frame
> pdf = sdf._to_pandas(pandasStructHandlingMode="row")
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
> line 1974, in _to_pandas
> return PandasConversionMixin._to_pandas(self, **kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/pandas/conversion.py",
> line 367, in _to_pandas
> batches = self._collect_as_arrow(
> ^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/pandas/conversion.py",
> line 537, in _collect_as_arrow
> with unwrap_spark_exception():
> File "/opt/conda/envs/lib/python3.11/contextlib.py", line 158, in __exit__
> self.gen.throw(typ, value, traceback)
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
> line 280, in unwrap_spark_exception
> raise converted from None
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
> line 410, in convert_column
> File "pyarrow/array.pxi", line 1341, in pyarrow.lib.Array.from_pandas
> File "pyarrow/array.pxi", line 414, in pyarrow.lib.array
> File "pyarrow/array.pxi", line 97, in pyarrow.lib._ndarray_to_array
> File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
> pyspark.errors.exceptions.captured.PythonException: An exception was thrown
> from the Python worker:
> Traceback (most recent call last):
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
> line 410, in convert_column
> return pa.Array.from_pandas(
> ^^^^^^^^^^^^^^^^^^^^^
> File "pyarrow/array.pxi", line 1341, in pyarrow.lib.Array.from_pandas
> File "pyarrow/array.pxi", line 414, in pyarrow.lib.array
> File "pyarrow/array.pxi", line 97, in pyarrow.lib._ndarray_to_array
> File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Integer value 9007199254740993 not in range:
> -9007199254740992 to 9007199254740992
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
> line 415, in convert_column
> return pa.Array.from_pandas(series, mask=mask).cast(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "pyarrow/array.pxi", line 1217, in pyarrow.lib.Array.cast
> File "/opt/conda/envs/lib/python3.11/site-packages/pyarrow/compute.py",
> line 415, in cast
> return call_function("cast", [arr], options, memory_pool)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "pyarrow/_compute.pyx", line 610, in pyarrow._compute.call_function
> File "pyarrow/_compute.pyx", line 405, in pyarrow._compute.Function.call
> File "pyarrow/error.pxi", line 155, in
> pyarrow.lib.pyarrow_internal_check_status
> File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Integer value 9007199254740993 not in range:
> -9007199254740992 to 9007199254740992
> The above exception was the direct cause of the following exception:
> Traceback (most recent call last):
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py",
> line 3622, in main
> process()
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py",
> line 3612, in process
> serializer.dump_stream(out_iter, outfile)
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 546, in dump_stream
> return ArrowStreamSerializer.dump_stream(self, batches, stream)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 146, in dump_stream
> for batch in iterator:
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 168, in _write_stream_start
> first = next(batch_iterator, None)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 543, in <genexpr>
> (create_batch(_normalize_packed(packed)) for packed in iterator),
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 529, in create_batch
> return PandasToArrowConversion.convert(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
> line 442, in convert
> arrays = [convert_column(col, field) for col, field in zip(columns,
> schema.fields)]
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
> line 442, in <listcomp>
> arrays = [convert_column(col, field) for col, field in zip(columns,
> schema.fields)]
> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
> line 440, in convert_column
> raise PySparkValueError(error_msg) from e
> pyspark.errors.exceptions.base.PySparkValueError: Failed to convert the value
> of the column '_0' with type 'int64' to Arrow type 'double'. It can be caused
> by overflows or other unsafe conversions warned by Arrow. Arrow safe type
> check can be disabled by using SQL config
> `spark.sql.execution.pandas.convertToArrowArraySafely`.
> {code}
> *Expected behavior*
> np.lcm applied to two Series containing 9007199254740993 should return an
> integer Series containing 9007199254740993, matching Pandas without an unsafe
> conversion to double.
> *Actual behavior*
> The computation raises pyspark.errors.exceptions.captured.PythonException.
> Its worker traceback contains pyarrow.lib.ArrowInvalid: Integer value
> 9007199254740993 not in range: -9007199254740992 to 9007199254740992,
> followed by a failure to convert an int64 column to Arrow double.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]