[
https://issues.apache.org/jira/browse/SPARK-50391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-50391:
---------------------------------
Summary: Support DataFrame conversion to table argument in Spark Classic
(was: DataFrame conversion to table argument)
> Support DataFrame conversion to table argument in Spark Classic
> ---------------------------------------------------------------
>
> Key: SPARK-50391
> URL: https://issues.apache.org/jira/browse/SPARK-50391
> Project: Spark
> Issue Type: Umbrella
> Components: Connect, PySpark, SQL
> Affects Versions: 4.0.0
> Reporter: Xinrong Meng
> Priority: Major
>
> Table-Valued Functions (TVFs) and User-Defined Table Functions (UDTFs) are
> widely used in Spark workflows. These functions often require a table
> argument, which Spark internally represents as a Catalyst expression. While
> Spark SQL supports constructs like TABLE(<query>) for this purpose, **there
> is no direct API in PySpark or Scala to convert a DataFrame into a table
> argument**. So we propose to support DataFrame conversion to table arguments
> (in Spark Classic first), and enable UDTFs to accept table arguments in both
> PySpark and Scala..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
