Ahmed Kamal` created SPARK-26393:
------------------------------------
Summary: Different behaviors of date_add when calling it inside
expr
Key: SPARK-26393
URL: https://issues.apache.org/jira/browse/SPARK-26393
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.3.2
Reporter: Ahmed Kamal`
When Calling date_add from pyspark.sql.functions directly without using expr,
like this :
{code:java}
df.withColumn("added", F.date_add(F.to_date(F.lit('1998-9-26')),
F.col('days'))).toPandas(){code}
It will raise Error : `TypeError: Column is not iterable`
because it only taking a number not a column
but when i try to use it inside an expr, like this :
{code:java}
df.withColumn("added", F.expr("date_add(to_date('1998-9-26'),
days)")).toPandas()
{code}
it will work fine.
Shouldn't it behave the same way ?
and i thin its logical to accept a column here as well.
A python Notebook to demonstrate :
https://gist.github.com/AhmedKamal20/fec10337e815baa44f115d307e3b07eb
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
