[ https://issues.apache.org/jira/browse/SPARK-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391851#comment-14391851 ]
Josh Rosen commented on SPARK-6553: ----------------------------------- This was fixed by https://github.com/apache/spark/pull/5206 for 1.3.1 and 1.4.0. > Support for functools.partial as UserDefinedFunction > ---------------------------------------------------- > > Key: SPARK-6553 > URL: https://issues.apache.org/jira/browse/SPARK-6553 > Project: Spark > Issue Type: Improvement > Components: PySpark > Affects Versions: 1.3.0 > Reporter: Kalle Jepsen > Assignee: Kalle Jepsen > Labels: features > Fix For: 1.3.1, 1.4.0 > > > Currently {{functools.partial}} s cannot be used as {{UserDefinedFunction}} s > for {{DataFrame}} s, as the {{\_\_name\_\_}} attribute does not exist. > Passing a {{functools.partial}} object will raise an Exception at > https://github.com/apache/spark/blob/master/python/pyspark/sql/functions.py#L126. > > {{functools.partial}} is very widely used and should probably be supported, > despite its lack of a {{\_\_name\_\_}}. > My suggestion is to use {{f.\_\_repr\_\_()}} instead, or check with > {{hasattr(f, '\_\_name\_\_)}} and use {{\_\_class\_\_}} if {{False}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org