[
https://issues.apache.org/jira/browse/SPARK-48555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Serruya updated SPARK-48555:
--------------------------------
Description:
Currently, several SQL functions accept both native types and Columns, but only
accept native types in their scala/python APIs:
* array_remove (works in SQL, scala, not in python)
* array_position(works in SQL, scala, not in python)
* map_contains_key (works in SQL, scala, not in python)
* substring (works only in SQL)
For example, this is possible in SQL:
{code:python}
spark.sql("select array_remove(col1, col2) from values(array(1,2,3), 2)")
{code}
But not in python:
{code:python}
df.select(F.array_remove(F.col("col1"), F.col("col2"))
{code}
was:
Currently, several SQL functions accept both native types and Columns, but only
accept native types in their scala/python APIs:
* array_remove (works in SQL, scala, not in python)
* array_position(works in SQL, scala, not in python)
* map_contains_key (works in SQL, scala, not in python)
* substring (works only in SQL)
For example, this is possible in SQL:
{code:python}
spark.sql("select array_remove(col1, col2) from values(array(1,2,3), 2)")
{code}
{code:python}
df.select(F.array_remove(F.col("col1"), F.col("col2"))
{code}
> Support Column type for several SQL functions in scala and python
> -----------------------------------------------------------------
>
> Key: SPARK-48555
> URL: https://issues.apache.org/jira/browse/SPARK-48555
> Project: Spark
> Issue Type: New Feature
> Components: Connect, PySpark, Spark Core
> Affects Versions: 3.5.1
> Reporter: Ron Serruya
> Priority: Minor
>
> Currently, several SQL functions accept both native types and Columns, but
> only accept native types in their scala/python APIs:
> * array_remove (works in SQL, scala, not in python)
> * array_position(works in SQL, scala, not in python)
> * map_contains_key (works in SQL, scala, not in python)
> * substring (works only in SQL)
> For example, this is possible in SQL:
> {code:python}
> spark.sql("select array_remove(col1, col2) from values(array(1,2,3), 2)")
> {code}
> But not in python:
> {code:python}
> df.select(F.array_remove(F.col("col1"), F.col("col2"))
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
