[ https://issues.apache.org/jira/browse/SPARK-26393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Kamal` updated SPARK-26393: --------------------------------- Description: When Calling date_add from pyspark.sql.functions directly without using expr, like this : {code:java} df.withColumn("added", F.date_add(F.to_date(F.lit('1998-9-26')), F.col('days'))).toPandas(){code} It will raise Error : `TypeError: Column is not iterable` because it only taking a number not a column but when i try to use it inside an expr, like this : {code:java} df.withColumn("added", F.expr("date_add(to_date('1998-9-26'), days)")).toPandas(){code} It will work fine. Shouldn't it behave the same way ? and i think its logical to accept a column here as well. A python Notebook to demonstrate : [https://gist.github.com/AhmedKamal20/fec10337e815baa44f115d307e3b07eb] was: When Calling date_add from pyspark.sql.functions directly without using expr, like this : {code:java} df.withColumn("added", F.date_add(F.to_date(F.lit('1998-9-26')), F.col('days'))).toPandas(){code} It will raise Error : `TypeError: Column is not iterable` because it only taking a number not a column but when i try to use it inside an expr, like this : {code:java} df.withColumn("added", F.expr("date_add(to_date('1998-9-26'), days)")).toPandas() {code} it will work fine. Shouldn't it behave the same way ? and i thin its logical to accept a column here as well. A python Notebook to demonstrate : https://gist.github.com/AhmedKamal20/fec10337e815baa44f115d307e3b07eb > Different behaviors of date_add when calling it inside expr > ----------------------------------------------------------- > > Key: SPARK-26393 > URL: https://issues.apache.org/jira/browse/SPARK-26393 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.3.2 > Reporter: Ahmed Kamal` > Priority: Minor > > When Calling date_add from pyspark.sql.functions directly without using expr, > like this : > {code:java} > df.withColumn("added", F.date_add(F.to_date(F.lit('1998-9-26')), > F.col('days'))).toPandas(){code} > It will raise Error : `TypeError: Column is not iterable` > because it only taking a number not a column > but when i try to use it inside an expr, like this : > {code:java} > df.withColumn("added", F.expr("date_add(to_date('1998-9-26'), > days)")).toPandas(){code} > It will work fine. > Shouldn't it behave the same way ? > and i think its logical to accept a column here as well. > A python Notebook to demonstrate : > [https://gist.github.com/AhmedKamal20/fec10337e815baa44f115d307e3b07eb] -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org