[
https://issues.apache.org/jira/browse/SPARK-61007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61007:
------------------------------------
Assignee: Hyukjin Kwon
> Support the subquery expressions of scalar, exists, isin and asTable of the
> Python client as PySpark does
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61007
> URL: https://issues.apache.org/jira/browse/SPARK-61007
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark 4.3.0 Connect, {{DataFrame.scalar()}} and {{DataFrame.exists()}}
> return the Columns of a scalar and an EXISTS subquery of the DataFrame,
> {{Column.isin(df)}} that of an IN subquery, and {{DataFrame.asTable()}} a
> {{TableArg}} that a UDTF call takes, partitioned and ordered as given:
> {{SubqueryExpression}}s of the plan of the DataFrame, which a plan sends in a
> {{WithRelations}} with the plans that it refers to. So
> {{t1.where(t2.where(col("t1.a").outer() == col("t2.a")).exists())}} and
> {{t1.select(spark.range(1).select(max("id")).scalar())}} are correlated and
> scalar subqueries, and {{udtf_call(df.asTable().partitionBy("id"))}} calls a
> UDTF with the rows of {{df}}.
> The Rust-backed Python client's {{scalar()}} runs the query and returns its
> first value, and {{exists()}} returns a bool, so the subqueries above fail;
> {{isin(df)}} raises {{UNSUPPORTED_DATA_TYPE}}, and so does a UDTF call with a
> {{TableArg}}. Support the subquery expressions as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
