[
https://issues.apache.org/jira/browse/SPARK-60982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60982:
---------------------------------
Description:
In PySpark, {{lit}} of a list builds an array of the literals of its elements:
{{F.lit([1, 2])}} is {{array(1, 2)}}, and a list with a Column raises
{{PySparkValueError}} {{COLUMN_IN_LIST}}. A one-dimensional NumPy array becomes
such an array cast to the array type of its dtype, and a dtype without a Spark
type raises {{UNSUPPORTED_NUMPY_ARRAY_SCALAR}}. The Rust-backed Python client
raises {{TypeError: Expected Column or scalar ...; got list}} for a list. Build
the array as PySpark does.
As PySpark's functions take such a value with {{lit}}, a list in a function's
argument is an array too: {{F.array_contains("a", [1])}} is {{array_contains(a,
array(1))}}, and {{F.when(c, [1, 2])}} has the value {{array(1, 2)}}.
{{array}}, {{create_map}}, {{map_concat}} and {{struct}} take a single list,
set or tuple as their columns, as in {{F.array(["a", "b"])}}, which the client
rejects. A value of another type raises {{PySparkTypeError}}
{{UNSUPPORTED_DATA_TYPE}}, where the client raises {{TypeError}}, and the
client converts a NumPy array of one element to that element, so
{{F.lit(np.array([1.5]))}} is the scalar {{1.5}} rather than an array.
was:
In PySpark, {{lit}} of a list builds an array of the literals of its elements:
{{F.lit([1, 2])}} is {{array(1, 2)}}, and a list with a Column raises
{{PySparkValueError}} {{COLUMN_IN_LIST}}. A one-dimensional NumPy array becomes
such an array cast to the array type of its dtype, and a dtype without a Spark
type raises {{UNSUPPORTED_NUMPY_ARRAY_SCALAR}}. The Rust-backed Python client
raises {{TypeError: Expected Column or scalar ...; got list}} for a list. Build
the array as PySpark does.
> Build an array of a list or a NumPy array in lit() in Spark Connect Rust as
> PySpark does
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-60982
> URL: https://issues.apache.org/jira/browse/SPARK-60982
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, {{lit}} of a list builds an array of the literals of its
> elements: {{F.lit([1, 2])}} is {{array(1, 2)}}, and a list with a Column
> raises {{PySparkValueError}} {{COLUMN_IN_LIST}}. A one-dimensional NumPy
> array becomes such an array cast to the array type of its dtype, and a dtype
> without a Spark type raises {{UNSUPPORTED_NUMPY_ARRAY_SCALAR}}. The
> Rust-backed Python client raises {{TypeError: Expected Column or scalar ...;
> got list}} for a list. Build the array as PySpark does.
> As PySpark's functions take such a value with {{lit}}, a list in a function's
> argument is an array too: {{F.array_contains("a", [1])}} is
> {{array_contains(a, array(1))}}, and {{F.when(c, [1, 2])}} has the value
> {{array(1, 2)}}. {{array}}, {{create_map}}, {{map_concat}} and {{struct}}
> take a single list, set or tuple as their columns, as in {{F.array(["a",
> "b"])}}, which the client rejects. A value of another type raises
> {{PySparkTypeError}} {{UNSUPPORTED_DATA_TYPE}}, where the client raises
> {{TypeError}}, and the client converts a NumPy array of one element to that
> element, so {{F.lit(np.array([1.5]))}} is the scalar {{1.5}} rather than an
> array.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
