[
https://issues.apache.org/jira/browse/SPARK-61562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144254#comment-18144254
]
Narendra Singh commented on SPARK-61562:
----------------------------------------
I've opened a PR that sends NumPy scalars to LiteralExpression in the Connect
Column operators, matching classic PySpark.
One design point for review: accept NumPy scalars (as classic does), or raise
UNSUPPORTED_DATA_TYPE instead of the AttributeError? The PR accepts them.
Unsigned NumPy types (e.g. np.uint8) are still unsupported by Connect literal
inference, which could be a follow-up. Happy to adjust the direction if
preferred.
> Spark Connect Column operators fail with NumPy scalar operands
> --------------------------------------------------------------
>
> Key: SPARK-61562
> URL: https://issues.apache.org/jira/browse/SPARK-61562
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.3.0, 5.0.0
> Reporter: Narendra Singh
> Priority: Major
> Labels: pull-request-available
>
> In Spark Connect, Column operators and methods raise an AttributeError when
> the other operand is a NumPy scalar, while classic PySpark accepts it.
> {code:python}
> import numpy as np
> from pyspark.sql import functions as F
> df = spark.range(3).select(F.col("id").alias("x"))
> df.select(F.col("x") + np.int64(1)).collect()
> # Classic: [Row((x + 1)=1), Row((x + 1)=2), Row((x + 1)=3)]
> # Connect: AttributeError: 'numpy.int64' object has no attribute '_expr'
> {code}
> The same happens for np.int32, np.float32 and np.bool_ operands of arithmetic
> and comparison operators, and for Column.between. np.float64 works only
> because it is a subclass of Python float. F.lit(np.int64(1)) already works in
> Connect.
> Cause: _bin_op in python/pyspark/sql/connect/column.py turns only built-in
> Python types (bool, int, float, str, date/time, Decimal) into literals and
> treats anything else as a Column, although LiteralExpression already supports
> NumPy scalars (np.generic).
> Proposed fix: also send NumPy scalars to LiteralExpression in _bin_op and the
> other Column methods that take literal values, and add tests that run on both
> classic and Connect.
> Related: SPARK-61473 records this AttributeError behavior of PySpark's
> _bin_op for the Spark Connect Rust client. Classic PySpark accepts NumPy
> scalars here, so this proposes accepting them in PySpark Connect too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]