Hyukjin Kwon created SPARK-61007:
------------------------------------
Summary: Support the subquery expressions of scalar, exists, isin
and asTable of the Python client as PySpark does
Key: SPARK-61007
URL: https://issues.apache.org/jira/browse/SPARK-61007
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark 4.3.0 Connect, {{DataFrame.scalar()}} and {{DataFrame.exists()}}
return the Columns of a scalar and an EXISTS subquery of the DataFrame,
{{Column.isin(df)}} that of an IN subquery, and {{DataFrame.asTable()}} a
{{TableArg}} that a UDTF call takes, partitioned and ordered as given:
{{SubqueryExpression}}s of the plan of the DataFrame, which a plan sends in a
{{WithRelations}} with the plans that it refers to. So
{{t1.where(t2.where(col("t1.a").outer() == col("t2.a")).exists())}} and
{{t1.select(spark.range(1).select(max("id")).scalar())}} are correlated and
scalar subqueries, and {{udtf_call(df.asTable().partitionBy("id"))}} calls a
UDTF with the rows of {{df}}.
The Rust-backed Python client's {{scalar()}} runs the query and returns its
first value, and {{exists()}} returns a bool, so the subqueries above fail;
{{isin(df)}} raises {{UNSUPPORTED_DATA_TYPE}}, and so does a UDTF call with a
{{TableArg}}. Support the subquery expressions as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
